Jul 28 20:45:37.603662 ip-10-0-133-14 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 20:45:37.603677 ip-10-0-133-14 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 20:45:37.603686 ip-10-0-133-14 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 20:45:37.603999 ip-10-0-133-14 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 20:45:47.834880 ip-10-0-133-14 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 20:45:47.834899 ip-10-0-133-14 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8726d43e51c548e5a1afa5f7a6b8d2d3 --
Jul 28 20:47:17.993452 ip-10-0-133-14 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 20:47:18.429703 ip-10-0-133-14 kubenswrapper[2537]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 20:47:18.429703 ip-10-0-133-14 kubenswrapper[2537]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 20:47:18.429703 ip-10-0-133-14 kubenswrapper[2537]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 20:47:18.429703 ip-10-0-133-14 kubenswrapper[2537]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 20:47:18.429703 ip-10-0-133-14 kubenswrapper[2537]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 20:47:18.432628 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.432531    2537 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 20:47:18.437582 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437562    2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:47:18.437582 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437578    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:47:18.437582 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437583    2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:47:18.437582 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437587    2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437591    2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437596    2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437607    2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437612    2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437616    2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437620    2537 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437625    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437629    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437633    2537 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437636    2537 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437641    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437646    2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437650    2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437654    2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437658    2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437663    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437668    2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437672    2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437676    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:47:18.437810 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437680    2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437684    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437688    2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437692    2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437696    2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437700    2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437704    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437709    2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437713    2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437718    2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437722    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437726    2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437730    2537 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437735    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437741    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437746    2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437751    2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437755    2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437760    2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437764    2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:47:18.438650 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437768    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437772    2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437776    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437780    2537 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437785    2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437789    2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437793    2537 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437798    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437802    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437806    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437810    2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437814    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437818    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437825    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437829    2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437833    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437838    2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437842    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437846    2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437852    2537 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:47:18.439511 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437859    2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437865    2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437870    2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437874    2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437880    2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437884    2537 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437890    2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437896    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437901    2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437905    2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437910    2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437914    2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437919    2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437925    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437930    2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437934    2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437939    2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437943    2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437947    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:47:18.440350 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437951    2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437958    2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437963    2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437971    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437975    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437980    2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437985    2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437990    2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437994    2537 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.437998    2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.438003    2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.438008    2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.438013    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.438018    2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.438022    2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.438029    2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.438035    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439203    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439213    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:47:18.441186 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439219    2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439225    2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439229    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439233    2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439238    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439242    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439246    2537 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439250    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439254    2537 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439259    2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439262    2537 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439270    2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439276    2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439281    2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439285    2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439290    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439295    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439299    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439303    2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439307    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:47:18.442015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439312    2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439316    2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439320    2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439325    2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439329    2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439333    2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439337    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439342    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439346    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439351    2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439355    2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439359    2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439364    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439368    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439372    2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439377    2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439380    2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439384    2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439389    2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439393    2537 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:47:18.442703 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439397    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439401    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439405    2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439409    2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439413    2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439419    2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439428    2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439435    2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439440    2537 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439444    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439449    2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439454    2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439459    2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439463    2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439467    2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439471    2537 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439476    2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439480    2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439484    2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:47:18.443277 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439488    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439493    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439497    2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439501    2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439505    2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439511    2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439514    2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439519    2537 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439523    2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439527    2537 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439531    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439535    2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439541    2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439545    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439550    2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439555    2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439559    2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439563    2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439568    2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439572    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:47:18.443745 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439576    2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439580    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439584    2537 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439588    2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439593    2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439597    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439601    2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439605    2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439610    2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439616    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439619    2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439624    2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439628    2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439632    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439637    2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439641    2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439645    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.439650    2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440284    2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440293    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:47:18.444373 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440298    2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440303    2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440307    2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440313    2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440317    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440323    2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440327    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440332    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440336    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440341    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440346    2537 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440351    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440355    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440359    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440364    2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440368    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440372    2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440376    2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440380    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440385    2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:47:18.444863 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440389    2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440393    2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440397    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440402    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440408    2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440412    2537 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440417    2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440421    2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440425    2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440430    2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440435    2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440439    2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440444    2537 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440448    2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440453    2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440457    2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440461    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440465    2537 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440469    2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440474    2537 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:47:18.445505 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440478    2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440483    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440487    2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440491    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440495    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440499    2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440503    2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440508    2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440512    2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440516    2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440521    2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440525    2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440529    2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440534    2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440538    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440542    2537 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440547    2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440551    2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440555    2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440560    2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:47:18.446015 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440564    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440569    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440573    2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440578    2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440582    2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440586    2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440590    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440595    2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440601    2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440607    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440611    2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440615    2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440619    2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440623    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440627    2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440631    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440635    2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440639    2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440644    2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440649    2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:47:18.446539 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440653    2537 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440658    2537 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440664    2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440672    2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440677    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440682    2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440686    2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440692    2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440699    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440705    2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440711    2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440715    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440720    2537 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440725    2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440730    2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440734    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.440739    2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440829    2537 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440839    2537 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 20:47:18.447099 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440850    2537 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440857    2537 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440863    2537 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440869    2537 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440875    2537 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440882    2537 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440887    2537 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440892    2537 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440898    2537 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440903    2537 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440908    2537 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440913    2537 flags.go:64] FLAG: --cgroup-root=""
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440918    2537 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440923    2537 flags.go:64] FLAG: --client-ca-file=""
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440928    2537 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440933    2537 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440939    2537 flags.go:64] FLAG: --cluster-domain=""
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440944    2537 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440949    2537 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440954    2537 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440960    2537 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440968    2537 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440973    2537 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 20:47:18.447583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440978    2537 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440983    2537 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440988    2537 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440993    2537 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.440998    2537 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441003    2537 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441008    2537 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441015    2537 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441020    2537 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441024    2537 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441028    2537 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441033    2537 flags.go:64] FLAG: --enable-server="true"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441038    2537 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441046    2537 flags.go:64] FLAG: --event-burst="100"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441051    2537 flags.go:64] FLAG: --event-qps="50"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441055    2537 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441060    2537 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441065    2537 flags.go:64] FLAG: --eviction-hard=""
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441071    2537 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441076    2537 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441080    2537 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441086    2537 flags.go:64] FLAG: --eviction-soft=""
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441090    2537 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441095    2537 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441100    2537 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 20:47:18.448186 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441104    2537 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441109    2537 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441113    2537 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441118    2537 flags.go:64] FLAG: --feature-gates=""
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441124    2537 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441149    2537 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441155    2537 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441160    2537 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441165    2537 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441170    2537 flags.go:64] FLAG: --help="false"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441174    2537 flags.go:64] FLAG: --hostname-override="ip-10-0-133-14.ec2.internal"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441181    2537 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441185    2537 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441190    2537 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441195    2537 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441201    2537 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441206    2537 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441211    2537 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441215    2537 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441220    2537 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441225    2537 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441229    2537 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441234    2537 flags.go:64] FLAG: --kube-reserved=""
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441239    2537 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 20:47:18.448777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441244    2537 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441249    2537 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441256    2537 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441261    2537 flags.go:64] FLAG: --lock-file=""
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441265    2537 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441270    2537 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441283    2537 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441292    2537 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441297    2537 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441301    2537 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441306    2537 flags.go:64] FLAG: --logging-format="text"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441311    2537 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441316    2537 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441322    2537 flags.go:64] FLAG: --manifest-url=""
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441327    2537 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441334    2537 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441339    2537 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441347    2537 flags.go:64] FLAG: --max-pods="110"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441352    2537 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441357    2537 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441363    2537 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441368    2537 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441373    2537 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441378    2537 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441382    2537 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 20:47:18.449378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441393    2537 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441398    2537 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441403    2537 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441408    2537 flags.go:64] FLAG: --pod-cidr=""
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441413    2537 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441421    2537 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441426    2537 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441431    2537 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441435    2537 flags.go:64] FLAG: --port="10250"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441442    2537 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441447    2537 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07bef7c6c2271133c"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441453    2537 flags.go:64] FLAG: --qos-reserved=""
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441458    2537 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441462    2537 flags.go:64] FLAG: --register-node="true"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441467    2537 flags.go:64] FLAG: --register-with-taints=""
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441473    2537 flags.go:64] FLAG: --registry-burst="10"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441478    2537 flags.go:64] FLAG: --registry-qps="5"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441482    2537 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441487    2537 flags.go:64] FLAG: --reserved-memory=""
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441493    2537 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441498    2537 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441506    2537 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441511    2537 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441516    2537 flags.go:64] FLAG: --runonce="false"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441520    2537 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 20:47:18.450125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441525    2537 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441530    2537 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441535    2537 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441540    2537 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441545    2537 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441550    2537 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441555    2537 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441560    2537 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441565    2537 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441569    2537 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441574    2537 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441585    2537 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441590    2537 flags.go:64] FLAG: --system-cgroups=""
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441595    2537 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441604    2537 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441608    2537 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441616    2537 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441622    2537 flags.go:64] FLAG: --tls-min-version=""
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441627    2537 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441631    2537 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441636    2537 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441641    2537 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441646    2537 flags.go:64] FLAG: --v="2"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441658    2537 flags.go:64] FLAG: --version="false"
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441665    2537 flags.go:64] FLAG: --vmodule=""
Jul 28 20:47:18.451064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441671    2537 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.441676    2537 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441856    2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441865    2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441870    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441875    2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441879    2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441886    2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441891    2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441897    2537 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441902    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441909    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441914    2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441918    2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441922    2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441927    2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441931    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441935    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441939    2537 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:47:18.451912 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441944    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441948    2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441952    2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441956    2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441962    2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441966    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441971    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441975    2537 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441979    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441984    2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441988    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441992    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.441996    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442001    2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442005    2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442009    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442014    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442018    2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442022    2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442026    2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:47:18.452444 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442030    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442034    2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442038    2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442042    2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442047    2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442055    2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442059    2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442063    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442067    2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442072    2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442076    2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442080    2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442084    2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442089    2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442093    2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442097    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442104    2537 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442108    2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442113    2537 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442117    2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:47:18.452928 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442121    2537 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442126    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442130    2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442151    2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442156    2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442161    2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442165    2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442169    2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442175    2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442179    2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442184    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442187    2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442192    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442196    2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442200    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442204    2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442209    2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442213    2537 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442218    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442222    2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:47:18.453435 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442226    2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442230    2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442234    2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442238    2537 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442243    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442247    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442251    2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442255    2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442261    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442268    2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442273    2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442278    2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442282    2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442286    2537 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442290    2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442294    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442299    2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442303    2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442309    2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:47:18.453927 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442314    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:47:18.454450 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442321    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:47:18.454450 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.442327    2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:47:18.454450 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.443083    2537 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 20:47:18.454450 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.452256    2537 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 20:47:18.454450 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.452271    2537 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 20:47:18.454450 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.452291    2537 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 20:47:18.454450 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.452297    2537 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 20:47:18.454450 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452352    2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:47:18.454450 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452356    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:47:18.454450 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452360    2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:47:18.454450 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452363    2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:47:18.454450 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452366    2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:47:18.454450 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452369    2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:47:18.454450 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452372    2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452375    2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452377    2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452380    2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452382    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452385    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452388    2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452391    2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452393    2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452396    2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452398    2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452401    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452404    2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452406    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452409    2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452412    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452415    2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452418    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452420    2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452423    2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:47:18.454801 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452426    2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452428    2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452431    2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452434    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452437    2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452440    2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452443    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452446    2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452449    2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452452    2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452455    2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452457    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452460    2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452463    2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452465    2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452468    2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452472    2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452476    2537 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452479    2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:47:18.455301 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452482    2537 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452485    2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452488    2537 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452491    2537 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452494    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452497    2537 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452501    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452503    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452506    2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452509    2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452512    2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452515    2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452517    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452520    2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452523    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452526    2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452528    2537 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452531    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452533    2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452536    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:47:18.455756 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452538    2537 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452541    2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452544    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452546    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452549    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452551    2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452554    2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452557    2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452560    2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452563    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452566    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452569    2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452571    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452574    2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452578    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452580    2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452583    2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452585    2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452588    2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:47:18.456258 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452590    2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:47:18.456702 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452593    2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:47:18.456702 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452595    2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:47:18.456702 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452597    2537 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:47:18.456702 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452600    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:47:18.456702 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452602    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:47:18.456702 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452605    2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:47:18.456702 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452608    2537 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:47:18.456702 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452611    2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:47:18.456702 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452615    2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:47:18.456702 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452618    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:47:18.456702 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452622    2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:47:18.456702 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452624    2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:47:18.456702 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452627    2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:47:18.456702 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452629    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:47:18.456702 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.452634    2537 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452746    2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452751    2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452754    2537 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452757    2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452760    2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452763    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452766    2537 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452769    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452772    2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452774    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452777    2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452779    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452784    2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452787    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452790    2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452793    2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452796    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452799    2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:47:18.457064 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452802    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452804    2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452807    2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452810    2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452812    2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452815    2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452817    2537 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452820    2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452823    2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452825    2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452828    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452830    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452833    2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452835    2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452838    2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452841    2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452843    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452845    2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452849    2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:47:18.457588 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452852    2537 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452855    2537 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452858    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452861    2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452864    2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452867    2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452869    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452872    2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452875    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452878    2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452881    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452883    2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452885    2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452888    2537 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452890    2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452893    2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452895    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452898    2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452900    2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452903    2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:47:18.458035 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452905    2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452908    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452910    2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452914    2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452917    2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452920    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452923    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452925    2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452928    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452930    2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452933    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452935    2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452938    2537 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452941    2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452944    2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452947    2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452949    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452952    2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452954    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:47:18.458589 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452957    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452959    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452962    2537 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452965    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452967    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452970    2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452972    2537 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452975    2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452977    2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452980    2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452982    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452985    2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452988    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452990    2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452993    2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452995    2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.452998    2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.453000    2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.453003    2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.453005    2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:47:18.459052 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.453008    2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:47:18.459624 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.453010    2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:47:18.459624 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:18.453012    2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:47:18.459624 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.453017    2537 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 20:47:18.459624 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.453121    2537 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 20:47:18.459624 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.458250    2537 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 20:47:18.459960 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.459948    2537 server.go:1019] "Starting client certificate rotation"
Jul 28 20:47:18.460068 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.460053    2537 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 20:47:18.460102 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.460090    2537 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 20:47:18.463072 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.463058    2537 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 20:47:18.465114 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.465099    2537 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 20:47:18.476559 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.476538    2537 log.go:25] "Validated CRI v1 runtime API"
Jul 28 20:47:18.479473 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.479456    2537 log.go:25] "Validated CRI v1 image API"
Jul 28 20:47:18.480288 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.480274    2537 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 20:47:18.483692 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.483670    2537 fs.go:135] Filesystem UUIDs: map[73f2cb16-a856-4667-b293-4ea7accf790d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e62ff0b0-b933-4407-9a86-cd6a802cd741:/dev/nvme0n1p4]
Jul 28 20:47:18.483763 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.483691    2537 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 20:47:18.491448 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.491311    2537 manager.go:216] Machine: {Timestamp:2026-07-28 20:47:18.489825981 +0000 UTC m=+0.371126849 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3132984 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2183ce0b93b086edf0116a4586b1fb SystemUUID:ec2183ce-0b93-b086-edf0-116a4586b1fb BootID:8726d43e-51c5-48e5-a1af-a5f7a6b8d2d3 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a1:68:a7:c7:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a1:68:a7:c7:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:2b:87:80:4f:72 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 20:47:18.491448 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.491434    2537 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 20:47:18.491581 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.491515    2537 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 20:47:18.491581 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.491550    2537 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 20:47:18.493463 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.493441    2537 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 20:47:18.493610 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.493466    2537 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-14.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 20:47:18.494302 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.494291    2537 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 20:47:18.494340 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.494305    2537 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 20:47:18.494340 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.494312    2537 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 20:47:18.494340 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.494327    2537 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 20:47:18.495588 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.495578    2537 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 20:47:18.498377 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.498366    2537 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 20:47:18.498485 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.498476    2537 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 20:47:18.498522 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.498504    2537 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 20:47:18.498522 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.498513    2537 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 20:47:18.498596 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.498523    2537 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 20:47:18.498596 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.498532    2537 kubelet.go:398] "Adding apiserver pod source"
Jul 28 20:47:18.498596 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.498540    2537 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 20:47:18.498889 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.498876    2537 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 20:47:18.498921 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.498897    2537 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 20:47:18.501919 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.501903    2537 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 20:47:18.504343 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.504329    2537 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 20:47:18.504401 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.504359    2537 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 20:47:18.506467 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.506454    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 20:47:18.506467 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.506470    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 20:47:18.506467 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.506476    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 20:47:18.506467 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.506481    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 20:47:18.506691 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.506493    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 20:47:18.506691 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.506499    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 20:47:18.506691 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.506507    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 20:47:18.506691 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.506512    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 20:47:18.506691 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.506524    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 20:47:18.506691 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.506529    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 20:47:18.506691 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.506544    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 20:47:18.506691 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.506549    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 20:47:18.506691 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.506562    2537 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 20:47:18.506691 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.506562    2537 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 20:47:18.507312 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.507298    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 20:47:18.507356 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.507314    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 20:47:18.509482 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.509466    2537 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-14.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 20:47:18.510490 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.510469    2537 server.go:1268] "Started kubelet"
Jul 28 20:47:18.510587 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.510512    2537 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 20:47:18.510587 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.510535    2537 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 20:47:18.510681 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.510596    2537 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 20:47:18.511349 ip-10-0-133-14 systemd[1]: Started Kubernetes Kubelet.
Jul 28 20:47:18.512815 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.511613    2537 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x9lpn"
Jul 28 20:47:18.512815 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.511640    2537 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 20:47:18.514593 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.513570    2537 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 20:47:18.516650 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.516543    2537 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 20:47:18.520123 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.520108    2537 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x9lpn"
Jul 28 20:47:18.525710 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.523722    2537 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-14.ec2.internal.18c68f18b0fbc984  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-14.ec2.internal,UID:ip-10-0-133-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-14.ec2.internal,},FirstTimestamp:2026-07-28 20:47:18.510258564 +0000 UTC m=+0.391559432,LastTimestamp:2026-07-28 20:47:18.510258564 +0000 UTC m=+0.391559432,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-14.ec2.internal,}"
Jul 28 20:47:18.525710 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.525337    2537 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 20:47:18.525710 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.525383    2537 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 20:47:18.525710 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.525439    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:18.526464 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.526442    2537 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 20:47:18.526540 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.526466    2537 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 20:47:18.526630 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.526617    2537 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 20:47:18.526696 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.526631    2537 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 20:47:18.526696 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.526654    2537 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 20:47:18.528444 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.528426    2537 factory.go:55] Registering systemd factory
Jul 28 20:47:18.528518 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.528447    2537 factory.go:223] Registration of the systemd container factory successfully
Jul 28 20:47:18.528670 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.528651    2537 factory.go:153] Registering CRI-O factory
Jul 28 20:47:18.528670 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.528669    2537 factory.go:223] Registration of the crio container factory successfully
Jul 28 20:47:18.528826 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.528734    2537 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 20:47:18.528826 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.528772    2537 factory.go:103] Registering Raw factory
Jul 28 20:47:18.528826 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.528803    2537 manager.go:1196] Started watching for new ooms in manager
Jul 28 20:47:18.529620 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.529604    2537 manager.go:349] Starting recovery of all containers
Jul 28 20:47:18.535723 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.535704    2537 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:47:18.536073 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.536026    2537 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 20:47:18.539227 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.539208    2537 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-14.ec2.internal\" not found" node="ip-10-0-133-14.ec2.internal"
Jul 28 20:47:18.541430 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.541405    2537 manager.go:354] Recovery completed
Jul 28 20:47:18.545975 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.545961    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:47:18.547723 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.547708    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:47:18.547804 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.547739    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:47:18.547804 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.547753    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:47:18.548339 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.548314    2537 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 20:47:18.548339 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.548330    2537 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 20:47:18.548339 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.548338    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:47:18.548475 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.548349    2537 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 20:47:18.549084 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.549069    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:47:18.549165 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.549098    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:47:18.549165 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.549108    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:47:18.625552 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.625526    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:18.687403 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.687360    2537 policy_none.go:49] "None policy: Start"
Jul 28 20:47:18.687403 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.687378    2537 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 20:47:18.687403 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.687390    2537 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 20:47:18.689409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.689396    2537 policy_none.go:47] "Start"
Jul 28 20:47:18.718854 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.718703    2537 manager.go:337] "Starting Device Plugin manager"
Jul 28 20:47:18.718974 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.718877    2537 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 20:47:18.718974 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.718892    2537 server.go:87] "Starting device plugin registration server"
Jul 28 20:47:18.719095 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.719082    2537 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 20:47:18.719156 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.719096    2537 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 20:47:18.719231 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.719207    2537 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 20:47:18.719306 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.719286    2537 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 20:47:18.719306 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.719295    2537 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 20:47:18.719812 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.719790    2537 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 20:47:18.719886 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.719847    2537 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:18.750591 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.750567    2537 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 20:47:18.750713 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.750613    2537 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 20:47:18.750713 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.750638    2537 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 20:47:18.750713 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.750677    2537 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 20:47:18.752632 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.752611    2537 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:47:18.819430 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.819392    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:47:18.820462 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.820446    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:47:18.820522 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.820477    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:47:18.820522 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.820487    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:47:18.820522 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.820512    2537 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-14.ec2.internal"
Jul 28 20:47:18.828854 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.828840    2537 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-14.ec2.internal"
Jul 28 20:47:18.828896 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.828861    2537 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-14.ec2.internal\": node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:18.838497 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.838477    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:18.850910 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.850882    2537 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-14.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal"]
Jul 28 20:47:18.850963 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.850946    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:47:18.856429 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.856412    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:47:18.856502 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.856440    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:47:18.856502 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.856450    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:47:18.858801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.858789    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:47:18.858934 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.858922    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-14.ec2.internal"
Jul 28 20:47:18.858968 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.858951    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:47:18.859537 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.859520    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:47:18.859537 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.859528    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:47:18.859673 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.859549    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:47:18.859673 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.859553    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:47:18.859673 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.859559    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:47:18.859673 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.859564    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:47:18.862377 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.862361    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal"
Jul 28 20:47:18.862453 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.862390    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:47:18.863055 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.863030    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:47:18.863151 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.863058    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:47:18.863151 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.863075    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:47:18.884677 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.884651    2537 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-14.ec2.internal\" not found" node="ip-10-0-133-14.ec2.internal"
Jul 28 20:47:18.889020 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.889005    2537 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-14.ec2.internal\" not found" node="ip-10-0-133-14.ec2.internal"
Jul 28 20:47:18.928561 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.928541    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/055a7a05e8fd2212ddcaf08826be7843-config\") pod \"kube-apiserver-proxy-ip-10-0-133-14.ec2.internal\" (UID: \"055a7a05e8fd2212ddcaf08826be7843\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-14.ec2.internal"
Jul 28 20:47:18.928619 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.928568    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95e38ebd473179562ecd1ee1c342a23a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal\" (UID: \"95e38ebd473179562ecd1ee1c342a23a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal"
Jul 28 20:47:18.928619 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:18.928586    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95e38ebd473179562ecd1ee1c342a23a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal\" (UID: \"95e38ebd473179562ecd1ee1c342a23a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal"
Jul 28 20:47:18.939063 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:18.939024    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:19.029100 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:19.029081    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95e38ebd473179562ecd1ee1c342a23a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal\" (UID: \"95e38ebd473179562ecd1ee1c342a23a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal"
Jul 28 20:47:19.029224 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:19.029107    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95e38ebd473179562ecd1ee1c342a23a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal\" (UID: \"95e38ebd473179562ecd1ee1c342a23a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal"
Jul 28 20:47:19.029224 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:19.029130    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/055a7a05e8fd2212ddcaf08826be7843-config\") pod \"kube-apiserver-proxy-ip-10-0-133-14.ec2.internal\" (UID: \"055a7a05e8fd2212ddcaf08826be7843\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-14.ec2.internal"
Jul 28 20:47:19.029224 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:19.029173    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95e38ebd473179562ecd1ee1c342a23a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal\" (UID: \"95e38ebd473179562ecd1ee1c342a23a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal"
Jul 28 20:47:19.029224 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:19.029187    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/055a7a05e8fd2212ddcaf08826be7843-config\") pod \"kube-apiserver-proxy-ip-10-0-133-14.ec2.internal\" (UID: \"055a7a05e8fd2212ddcaf08826be7843\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-14.ec2.internal"
Jul 28 20:47:19.029224 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:19.029199    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95e38ebd473179562ecd1ee1c342a23a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal\" (UID: \"95e38ebd473179562ecd1ee1c342a23a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal"
Jul 28 20:47:19.039142 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:19.039118    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:19.139700 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:19.139682    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:19.187885 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:19.187860    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-14.ec2.internal"
Jul 28 20:47:19.192896 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:19.192756    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal"
Jul 28 20:47:19.240266 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:19.240242    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:19.340773 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:19.340749    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:19.441292 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:19.441262    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:19.459454 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:19.459405    2537 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 20:47:19.459543 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:19.459524    2537 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 20:47:19.459581 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:19.459551    2537 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 20:47:19.496692 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:19.496671    2537 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:47:19.527774 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:19.527744    2537 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 20:42:19 +0000 UTC" deadline="2028-02-18 03:02:56.923748884 +0000 UTC"
Jul 28 20:47:19.527774 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:19.527774    2537 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13662h15m37.395978415s"
Jul 28 20:47:19.541832 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:19.541815    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:19.642380 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:19.642349    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:21.000470 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:21.000440    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055a7a05e8fd2212ddcaf08826be7843.slice/crio-16bdc7d31a579840e7c890e2ebc8b21072644cd22ed74f8e159dca7a66425d29 WatchSource:0}: Error finding container 16bdc7d31a579840e7c890e2ebc8b21072644cd22ed74f8e159dca7a66425d29: Status 404 returned error can't find the container with id 16bdc7d31a579840e7c890e2ebc8b21072644cd22ed74f8e159dca7a66425d29
Jul 28 20:47:21.000645 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:21.000633    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e38ebd473179562ecd1ee1c342a23a.slice/crio-b9c4c42b5fe6c3ca52b391ca39079783dee7fa37b7597942e83c9c701d2c7bd2 WatchSource:0}: Error finding container b9c4c42b5fe6c3ca52b391ca39079783dee7fa37b7597942e83c9c701d2c7bd2: Status 404 returned error can't find the container with id b9c4c42b5fe6c3ca52b391ca39079783dee7fa37b7597942e83c9c701d2c7bd2
Jul 28 20:47:21.004842 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.004827    2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:47:21.006506 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:21.006490    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:21.017835 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.017794    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-14.ec2.internal" event={"ID":"055a7a05e8fd2212ddcaf08826be7843","Type":"ContainerStarted","Data":"16bdc7d31a579840e7c890e2ebc8b21072644cd22ed74f8e159dca7a66425d29"}
Jul 28 20:47:21.018815 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.018792    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal" event={"ID":"95e38ebd473179562ecd1ee1c342a23a","Type":"ContainerStarted","Data":"b9c4c42b5fe6c3ca52b391ca39079783dee7fa37b7597942e83c9c701d2c7bd2"}
Jul 28 20:47:21.107365 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:21.107333    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:21.207903 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:21.207876    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:21.308712 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:21.308644    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-14.ec2.internal\" not found"
Jul 28 20:47:21.338683 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.338666    2537 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:47:21.390065 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.390043    2537 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal"
Jul 28 20:47:21.403803 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.403781    2537 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 20:47:21.404687 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.404670    2537 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-14.ec2.internal"
Jul 28 20:47:21.412439 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.412416    2537 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 20:47:21.527835 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.527809    2537 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:47:21.762993 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.762913    2537 apiserver.go:52] "Watching apiserver"
Jul 28 20:47:21.773140 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.773113    2537 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 20:47:21.773675 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.773613    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-tk54b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal","openshift-multus/multus-mkp2r","openshift-image-registry/node-ca-6vhdp","openshift-multus/multus-additional-cni-plugins-drhrl","openshift-multus/network-metrics-daemon-9qhw8","openshift-network-diagnostics/network-check-target-lw7l6","openshift-network-operator/iptables-alerter-6ws95","openshift-ovn-kubernetes/ovnkube-node-ldz9r","kube-system/global-pull-secret-syncer-xx9kc","kube-system/konnectivity-agent-d44fv","kube-system/kube-apiserver-proxy-ip-10-0-133-14.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t","openshift-cluster-node-tuning-operator/tuned-j9pmm"]
Jul 28 20:47:21.776772 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.776743    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:21.776885 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:21.776827    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:21.779181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.778726    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.780923 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.780902    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.782228 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.781457    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 20:47:21.782228 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.781647    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zzdfw\""
Jul 28 20:47:21.782228 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.781833    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 20:47:21.782228 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.782021    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 20:47:21.783461 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.783442    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8z4lv\""
Jul 28 20:47:21.783659 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.783646    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6vhdp"
Jul 28 20:47:21.783862 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.783837    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 20:47:21.784098 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.784083    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 20:47:21.784384 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.784364    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 20:47:21.784454 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.784407    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 20:47:21.785356 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.785336    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.786438 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.786232    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2mn7n\""
Jul 28 20:47:21.786438 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.786303    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 20:47:21.786438 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.786436    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 20:47:21.787022 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.787004    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 20:47:21.787611 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.787593    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:21.787693 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:21.787657    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:21.789612 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.789438    2537 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 20:47:21.790879 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.790839    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.792971 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.792947    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tk54b"
Jul 28 20:47:21.794940 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.794923    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 20:47:21.795080 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.795065    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 20:47:21.795135 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.795113    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8bzpb\""
Jul 28 20:47:21.795439 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.795419    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:21.795844 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.795827    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 20:47:21.795981 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.795966    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 20:47:21.796057 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.796038    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 20:47:21.796596 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.796578    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 20:47:21.796691 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.796622    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jk7m9\""
Jul 28 20:47:21.796879 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.796777    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 20:47:21.796879 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.796805    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 20:47:21.796879 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.796778    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 20:47:21.796879 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.796837    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9nkz8\""
Jul 28 20:47:21.797944 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.797926    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:21.798019 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:21.797998    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:21.798277 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.798145    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 20:47:21.799607 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.799587    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 20:47:21.799910 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.799892    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 20:47:21.800311 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.800289    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d44fv"
Jul 28 20:47:21.800475 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.800459    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7pqxd\""
Jul 28 20:47:21.801633 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.801612    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb0b5897-6159-4391-a86a-107312cea21e-system-cni-dir\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.801718 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.801648    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-run-systemd\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.801718 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.801690    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-run-ovn\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.801718 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.801713    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4mzq\" (UniqueName: \"kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq\") pod \"network-check-target-lw7l6\" (UID: \"59627965-db98-4b3f-82ce-b9621a1eef88\") " pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:21.801718 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.801737    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d984108-626d-4797-9164-85cc0c199bcc-cni-binary-copy\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.801954 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.801760    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.801954 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.801784    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-system-cni-dir\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.801954 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.801806    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-var-lib-cni-multus\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.801954 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.801830    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-var-lib-kubelet\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.802203 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.802028    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.802203 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.802060    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-run-netns\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.802203 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.802108    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-etc-selinux\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.802203 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.802132    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb0b5897-6159-4391-a86a-107312cea21e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.802203 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.802162    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-cni-bin\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.802203 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.802194    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de461c7c-9806-4aac-8afc-47e97696b0fb-ovnkube-config\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.802553 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.802241    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-registration-dir\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.802553 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.802268    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntjvg\" (UniqueName: \"kubernetes.io/projected/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-kube-api-access-ntjvg\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.802553 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.802400    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4421ad59-d310-4fce-b6f2-5518ede532fa-tmp-dir\") pod \"node-resolver-tk54b\" (UID: \"4421ad59-d310-4fce-b6f2-5518ede532fa\") " pod="openshift-dns/node-resolver-tk54b"
Jul 28 20:47:21.803041 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.803019    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6ws95"
Jul 28 20:47:21.803754 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.803733    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-multus-conf-dir\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.803835 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.803818    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-run-multus-certs\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.803921 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.803901    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/19ba6b7f-59fc-4288-aef0-a479c989e6d7-serviceca\") pod \"node-ca-6vhdp\" (UID: \"19ba6b7f-59fc-4288-aef0-a479c989e6d7\") " pod="openshift-image-registry/node-ca-6vhdp"
Jul 28 20:47:21.803976 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.803945    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb0b5897-6159-4391-a86a-107312cea21e-cnibin\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.803976 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.803960    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 20:47:21.804085 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804063    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4421ad59-d310-4fce-b6f2-5518ede532fa-hosts-file\") pod \"node-resolver-tk54b\" (UID: \"4421ad59-d310-4fce-b6f2-5518ede532fa\") " pod="openshift-dns/node-resolver-tk54b"
Jul 28 20:47:21.804180 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804123    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-os-release\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.804234 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804213    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-multus-socket-dir-parent\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.804290 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804252    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46tzc\" (UniqueName: \"kubernetes.io/projected/19ba6b7f-59fc-4288-aef0-a479c989e6d7-kube-api-access-46tzc\") pod \"node-ca-6vhdp\" (UID: \"19ba6b7f-59fc-4288-aef0-a479c989e6d7\") " pod="openshift-image-registry/node-ca-6vhdp"
Jul 28 20:47:21.804339 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804285    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de461c7c-9806-4aac-8afc-47e97696b0fb-env-overrides\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.804378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804347    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-run-k8s-cni-cncf-io\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.804426 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804393    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-socket-dir\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.804590 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804570    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19ba6b7f-59fc-4288-aef0-a479c989e6d7-host\") pod \"node-ca-6vhdp\" (UID: \"19ba6b7f-59fc-4288-aef0-a479c989e6d7\") " pod="openshift-image-registry/node-ca-6vhdp"
Jul 28 20:47:21.804654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804618    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-kubelet\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.804654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804648    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-run-netns\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.804753 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804671    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-etc-kubernetes\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.804753 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804699    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.804753 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804732    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-sys-fs\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.804930 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804772    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22mtp\" (UniqueName: \"kubernetes.io/projected/5e0b9b3f-4257-4183-90eb-1b7102f3606d-kube-api-access-22mtp\") pod \"network-metrics-daemon-9qhw8\" (UID: \"5e0b9b3f-4257-4183-90eb-1b7102f3606d\") " pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:21.804930 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804823    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de461c7c-9806-4aac-8afc-47e97696b0fb-ovnkube-script-lib\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.804930 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804881    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb0b5897-6159-4391-a86a-107312cea21e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.805086 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804926    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-systemd-units\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.805086 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.804966    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-etc-openvswitch\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.805086 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805002    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-node-log\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.805086 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805048    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb0b5897-6159-4391-a86a-107312cea21e-os-release\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.805086 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805074    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-run-openvswitch\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.805315 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805103    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-hostroot\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.805315 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805157    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb0b5897-6159-4391-a86a-107312cea21e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.805315 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805193    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de461c7c-9806-4aac-8afc-47e97696b0fb-ovn-node-metrics-cert\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.805315 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805218    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-device-dir\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.805315 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805249    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-cni-netd\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.805315 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805280    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64p8t\" (UniqueName: \"kubernetes.io/projected/de461c7c-9806-4aac-8afc-47e97696b0fb-kube-api-access-64p8t\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.805612 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805344    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2psc\" (UniqueName: \"kubernetes.io/projected/4421ad59-d310-4fce-b6f2-5518ede532fa-kube-api-access-r2psc\") pod \"node-resolver-tk54b\" (UID: \"4421ad59-d310-4fce-b6f2-5518ede532fa\") " pod="openshift-dns/node-resolver-tk54b"
Jul 28 20:47:21.805612 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805395    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-multus-cni-dir\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.805612 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805442    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb0b5897-6159-4391-a86a-107312cea21e-cni-binary-copy\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.805612 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805472    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-var-lib-openvswitch\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.805612 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805527    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-cnibin\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.805612 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805560    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9588\" (UniqueName: \"kubernetes.io/projected/7d984108-626d-4797-9164-85cc0c199bcc-kube-api-access-v9588\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.805612 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805592    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nvdt\" (UniqueName: \"kubernetes.io/projected/bb0b5897-6159-4391-a86a-107312cea21e-kube-api-access-4nvdt\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.805961 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805621    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs\") pod \"network-metrics-daemon-9qhw8\" (UID: \"5e0b9b3f-4257-4183-90eb-1b7102f3606d\") " pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:21.805961 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805650    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d984108-626d-4797-9164-85cc0c199bcc-multus-daemon-config\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.805961 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805670    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-slash\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.805961 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805709    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-log-socket\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.805961 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.805726    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-var-lib-cni-bin\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.806310 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.806096    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xhbxj\""
Jul 28 20:47:21.806873 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.806832    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 20:47:21.807784 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.807695    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lwhv4\""
Jul 28 20:47:21.808385 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.808365    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 20:47:21.808559 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.808375    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 20:47:21.808689 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.808601    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 20:47:21.812699 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.812670    2537 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 20:47:21.832272 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.832250    2537 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qqgn6"
Jul 28 20:47:21.844260 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.844240    2537 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qqgn6"
Jul 28 20:47:21.891545 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.891518    2537 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 20:47:21.906688 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.906666    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb0b5897-6159-4391-a86a-107312cea21e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.906819 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.906701    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-host\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:21.906819 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.906725    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-cni-bin\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.906819 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.906746    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de461c7c-9806-4aac-8afc-47e97696b0fb-ovnkube-config\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.906819 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.906768    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-registration-dir\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.906819 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.906790    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntjvg\" (UniqueName: \"kubernetes.io/projected/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-kube-api-access-ntjvg\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.906819 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.906812    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-lib-modules\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:21.907145 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.906836    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/415a6e79-19ba-4beb-ab92-8b2b739d63a4-dbus\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:21.907145 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.906875    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65bbeae6-101e-450a-bdcc-6d3c9684dad5-konnectivity-ca\") pod \"konnectivity-agent-d44fv\" (UID: \"65bbeae6-101e-450a-bdcc-6d3c9684dad5\") " pod="kube-system/konnectivity-agent-d44fv"
Jul 28 20:47:21.907145 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.906902    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4421ad59-d310-4fce-b6f2-5518ede532fa-tmp-dir\") pod \"node-resolver-tk54b\" (UID: \"4421ad59-d310-4fce-b6f2-5518ede532fa\") " pod="openshift-dns/node-resolver-tk54b"
Jul 28 20:47:21.907145 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.906924    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-multus-conf-dir\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.907145 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.906949    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-run-multus-certs\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.907145 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.906973    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/19ba6b7f-59fc-4288-aef0-a479c989e6d7-serviceca\") pod \"node-ca-6vhdp\" (UID: \"19ba6b7f-59fc-4288-aef0-a479c989e6d7\") " pod="openshift-image-registry/node-ca-6vhdp"
Jul 28 20:47:21.907145 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.906998    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb0b5897-6159-4391-a86a-107312cea21e-cnibin\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.907145 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907021    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-var-lib-kubelet\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:21.907145 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907044    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23f7f8e9-e2f3-40d5-b9d5-b86176627766-tmp\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:21.907145 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907074    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4421ad59-d310-4fce-b6f2-5518ede532fa-hosts-file\") pod \"node-resolver-tk54b\" (UID: \"4421ad59-d310-4fce-b6f2-5518ede532fa\") " pod="openshift-dns/node-resolver-tk54b"
Jul 28 20:47:21.907145 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907100    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-os-release\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.907145 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907126    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-multus-socket-dir-parent\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907149    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46tzc\" (UniqueName: \"kubernetes.io/projected/19ba6b7f-59fc-4288-aef0-a479c989e6d7-kube-api-access-46tzc\") pod \"node-ca-6vhdp\" (UID: \"19ba6b7f-59fc-4288-aef0-a479c989e6d7\") " pod="openshift-image-registry/node-ca-6vhdp"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907175    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9224c415-9d67-4c55-9d53-4a06b3a5a44a-iptables-alerter-script\") pod \"iptables-alerter-6ws95\" (UID: \"9224c415-9d67-4c55-9d53-4a06b3a5a44a\") " pod="openshift-network-operator/iptables-alerter-6ws95"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907202    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de461c7c-9806-4aac-8afc-47e97696b0fb-env-overrides\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907228    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-run-k8s-cni-cncf-io\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907290    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-run-k8s-cni-cncf-io\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907294    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-multus-conf-dir\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907332    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-cni-bin\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907351    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-socket-dir\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907398    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-registration-dir\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907398    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19ba6b7f-59fc-4288-aef0-a479c989e6d7-host\") pod \"node-ca-6vhdp\" (UID: \"19ba6b7f-59fc-4288-aef0-a479c989e6d7\") " pod="openshift-image-registry/node-ca-6vhdp"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907436    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19ba6b7f-59fc-4288-aef0-a479c989e6d7-host\") pod \"node-ca-6vhdp\" (UID: \"19ba6b7f-59fc-4288-aef0-a479c989e6d7\") " pod="openshift-image-registry/node-ca-6vhdp"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907441    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-kubelet\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907478    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-run-netns\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907481    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-socket-dir\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907489    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-kubelet\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907516    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-etc-kubernetes\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907544    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.907701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907549    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-os-release\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907572    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-sys-fs\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907518    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-run-netns\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907618    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4421ad59-d310-4fce-b6f2-5518ede532fa-hosts-file\") pod \"node-resolver-tk54b\" (UID: \"4421ad59-d310-4fce-b6f2-5518ede532fa\") " pod="openshift-dns/node-resolver-tk54b"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907630    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4421ad59-d310-4fce-b6f2-5518ede532fa-tmp-dir\") pod \"node-resolver-tk54b\" (UID: \"4421ad59-d310-4fce-b6f2-5518ede532fa\") " pod="openshift-dns/node-resolver-tk54b"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907708    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-sys-fs\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907733    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-modprobe-d\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907758    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9224c415-9d67-4c55-9d53-4a06b3a5a44a-host-slash\") pod \"iptables-alerter-6ws95\" (UID: \"9224c415-9d67-4c55-9d53-4a06b3a5a44a\") " pod="openshift-network-operator/iptables-alerter-6ws95"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907777    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22mtp\" (UniqueName: \"kubernetes.io/projected/5e0b9b3f-4257-4183-90eb-1b7102f3606d-kube-api-access-22mtp\") pod \"network-metrics-daemon-9qhw8\" (UID: \"5e0b9b3f-4257-4183-90eb-1b7102f3606d\") " pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907794    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de461c7c-9806-4aac-8afc-47e97696b0fb-ovnkube-script-lib\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907809    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb0b5897-6159-4391-a86a-107312cea21e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907830    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-tuned\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907864    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-systemd-units\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907882    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-etc-openvswitch\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907901    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-node-log\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907921    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb0b5897-6159-4391-a86a-107312cea21e-os-release\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907936    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj2p5\" (UniqueName: \"kubernetes.io/projected/23f7f8e9-e2f3-40d5-b9d5-b86176627766-kube-api-access-fj2p5\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:21.908609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907952    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-run-openvswitch\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907967    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-hostroot\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907971    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/19ba6b7f-59fc-4288-aef0-a479c989e6d7-serviceca\") pod \"node-ca-6vhdp\" (UID: \"19ba6b7f-59fc-4288-aef0-a479c989e6d7\") " pod="openshift-image-registry/node-ca-6vhdp"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.907988    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb0b5897-6159-4391-a86a-107312cea21e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908039    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-systemd-units\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908049    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de461c7c-9806-4aac-8afc-47e97696b0fb-ovn-node-metrics-cert\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908073    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-device-dir\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908078    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-etc-kubernetes\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908093    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-cni-netd\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908097    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de461c7c-9806-4aac-8afc-47e97696b0fb-env-overrides\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908113    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64p8t\" (UniqueName: \"kubernetes.io/projected/de461c7c-9806-4aac-8afc-47e97696b0fb-kube-api-access-64p8t\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908129    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2psc\" (UniqueName: \"kubernetes.io/projected/4421ad59-d310-4fce-b6f2-5518ede532fa-kube-api-access-r2psc\") pod \"node-resolver-tk54b\" (UID: \"4421ad59-d310-4fce-b6f2-5518ede532fa\") " pod="openshift-dns/node-resolver-tk54b"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908157    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de461c7c-9806-4aac-8afc-47e97696b0fb-ovnkube-config\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908166    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-multus-cni-dir\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908188    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-multus-socket-dir-parent\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908216    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-run-multus-certs\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908214    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb0b5897-6159-4391-a86a-107312cea21e-cni-binary-copy\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.909409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908246    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-sysconfig\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908254    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-multus-cni-dir\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908262    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-sys\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908282    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-run-openvswitch\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908287    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-var-lib-openvswitch\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908314    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-cnibin\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908366    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-hostroot\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908408    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-cni-netd\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908399    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb0b5897-6159-4391-a86a-107312cea21e-os-release\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908540    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de461c7c-9806-4aac-8afc-47e97696b0fb-ovnkube-script-lib\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908610    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-device-dir\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908700    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908700    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-etc-openvswitch\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908735    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-var-lib-openvswitch\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908749    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb0b5897-6159-4391-a86a-107312cea21e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908789    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb0b5897-6159-4391-a86a-107312cea21e-cnibin\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908840    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb0b5897-6159-4391-a86a-107312cea21e-cni-binary-copy\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908836    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-cnibin\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.910181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908917    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9588\" (UniqueName: \"kubernetes.io/projected/7d984108-626d-4797-9164-85cc0c199bcc-kube-api-access-v9588\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908972    2537 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.909245    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-node-log\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.908980    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nvdt\" (UniqueName: \"kubernetes.io/projected/bb0b5897-6159-4391-a86a-107312cea21e-kube-api-access-4nvdt\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.909327    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-kubernetes\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.909339    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb0b5897-6159-4391-a86a-107312cea21e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.909365    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.909373    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb0b5897-6159-4391-a86a-107312cea21e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.909397    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs\") pod \"network-metrics-daemon-9qhw8\" (UID: \"5e0b9b3f-4257-4183-90eb-1b7102f3606d\") " pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:21.909496    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:21.909632    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs podName:5e0b9b3f-4257-4183-90eb-1b7102f3606d nodeName:}" failed. No retries permitted until 2026-07-28 20:47:22.40959904 +0000 UTC m=+3.027180938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs") pod "network-metrics-daemon-9qhw8" (UID: "5e0b9b3f-4257-4183-90eb-1b7102f3606d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.909896    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d984108-626d-4797-9164-85cc0c199bcc-multus-daemon-config\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.909918    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-slash\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.909932    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-log-socket\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.909955    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-var-lib-cni-bin\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.909978    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb0b5897-6159-4391-a86a-107312cea21e-system-cni-dir\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910003    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-systemd\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:21.911018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910013    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d984108-626d-4797-9164-85cc0c199bcc-multus-daemon-config\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910026    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-run\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910053    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-run-systemd\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910071    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-log-socket\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910078    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-run-ovn\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910003    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-slash\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910095    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb0b5897-6159-4391-a86a-107312cea21e-system-cni-dir\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910044    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-var-lib-cni-bin\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910103    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mzq\" (UniqueName: \"kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq\") pod \"network-check-target-lw7l6\" (UID: \"59627965-db98-4b3f-82ce-b9621a1eef88\") " pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910128    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-run-systemd\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910142    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-run-ovn\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910137    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d984108-626d-4797-9164-85cc0c199bcc-cni-binary-copy\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910174    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-sysctl-d\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910199    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65bbeae6-101e-450a-bdcc-6d3c9684dad5-agent-certs\") pod \"konnectivity-agent-d44fv\" (UID: \"65bbeae6-101e-450a-bdcc-6d3c9684dad5\") " pod="kube-system/konnectivity-agent-d44fv"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910223    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7k4t\" (UniqueName: \"kubernetes.io/projected/9224c415-9d67-4c55-9d53-4a06b3a5a44a-kube-api-access-z7k4t\") pod \"iptables-alerter-6ws95\" (UID: \"9224c415-9d67-4c55-9d53-4a06b3a5a44a\") " pod="openshift-network-operator/iptables-alerter-6ws95"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910248    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910296    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-system-cni-dir\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.911801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910322    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.912654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910328    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-var-lib-cni-multus\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.912654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910360    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-system-cni-dir\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.912654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910351    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-var-lib-kubelet\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.912654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910387    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-var-lib-cni-multus\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.912654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910410    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-sysctl-conf\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:21.912654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910436    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/415a6e79-19ba-4beb-ab92-8b2b739d63a4-kubelet-config\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:21.912654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910411    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-var-lib-kubelet\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.912654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910500    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.912654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910527    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-run-netns\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.912654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910584    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-etc-selinux\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.912654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910621    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d984108-626d-4797-9164-85cc0c199bcc-host-run-netns\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.912654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910589    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de461c7c-9806-4aac-8afc-47e97696b0fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.912654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910649    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d984108-626d-4797-9164-85cc0c199bcc-cni-binary-copy\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.912654 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.910677    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-etc-selinux\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.913520 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.913493    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de461c7c-9806-4aac-8afc-47e97696b0fb-ovn-node-metrics-cert\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.921885 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.921811    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22mtp\" (UniqueName: \"kubernetes.io/projected/5e0b9b3f-4257-4183-90eb-1b7102f3606d-kube-api-access-22mtp\") pod \"network-metrics-daemon-9qhw8\" (UID: \"5e0b9b3f-4257-4183-90eb-1b7102f3606d\") " pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:21.927215 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:21.927193    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:47:21.927315 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:21.927219    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:47:21.927315 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:21.927233    2537 projected.go:196] Error preparing data for projected volume kube-api-access-m4mzq for pod openshift-network-diagnostics/network-check-target-lw7l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:21.930091 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:21.930071    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq podName:59627965-db98-4b3f-82ce-b9621a1eef88 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:22.430044447 +0000 UTC m=+3.047626333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m4mzq" (UniqueName: "kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq") pod "network-check-target-lw7l6" (UID: "59627965-db98-4b3f-82ce-b9621a1eef88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:21.930806 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.930761    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9588\" (UniqueName: \"kubernetes.io/projected/7d984108-626d-4797-9164-85cc0c199bcc-kube-api-access-v9588\") pod \"multus-mkp2r\" (UID: \"7d984108-626d-4797-9164-85cc0c199bcc\") " pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:21.931166 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.931149    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46tzc\" (UniqueName: \"kubernetes.io/projected/19ba6b7f-59fc-4288-aef0-a479c989e6d7-kube-api-access-46tzc\") pod \"node-ca-6vhdp\" (UID: \"19ba6b7f-59fc-4288-aef0-a479c989e6d7\") " pod="openshift-image-registry/node-ca-6vhdp"
Jul 28 20:47:21.931405 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.931390    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntjvg\" (UniqueName: \"kubernetes.io/projected/8b4a9a4e-beb8-47a6-ad56-a375b0412c2a-kube-api-access-ntjvg\") pod \"aws-ebs-csi-driver-node-l886t\" (UID: \"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:21.931632 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.931605    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64p8t\" (UniqueName: \"kubernetes.io/projected/de461c7c-9806-4aac-8afc-47e97696b0fb-kube-api-access-64p8t\") pod \"ovnkube-node-ldz9r\" (UID: \"de461c7c-9806-4aac-8afc-47e97696b0fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:21.932006 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.931921    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2psc\" (UniqueName: \"kubernetes.io/projected/4421ad59-d310-4fce-b6f2-5518ede532fa-kube-api-access-r2psc\") pod \"node-resolver-tk54b\" (UID: \"4421ad59-d310-4fce-b6f2-5518ede532fa\") " pod="openshift-dns/node-resolver-tk54b"
Jul 28 20:47:21.932203 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.932176    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nvdt\" (UniqueName: \"kubernetes.io/projected/bb0b5897-6159-4391-a86a-107312cea21e-kube-api-access-4nvdt\") pod \"multus-additional-cni-plugins-drhrl\" (UID: \"bb0b5897-6159-4391-a86a-107312cea21e\") " pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:21.956182 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:21.956159    2537 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:47:22.010926 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.010886    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-modprobe-d\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.011060 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.010930    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9224c415-9d67-4c55-9d53-4a06b3a5a44a-host-slash\") pod \"iptables-alerter-6ws95\" (UID: \"9224c415-9d67-4c55-9d53-4a06b3a5a44a\") " pod="openshift-network-operator/iptables-alerter-6ws95"
Jul 28 20:47:22.011060 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011017    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-tuned\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.011060 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011050    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-modprobe-d\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.011228 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011074    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9224c415-9d67-4c55-9d53-4a06b3a5a44a-host-slash\") pod \"iptables-alerter-6ws95\" (UID: \"9224c415-9d67-4c55-9d53-4a06b3a5a44a\") " pod="openshift-network-operator/iptables-alerter-6ws95"
Jul 28 20:47:22.011228 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011114    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj2p5\" (UniqueName: \"kubernetes.io/projected/23f7f8e9-e2f3-40d5-b9d5-b86176627766-kube-api-access-fj2p5\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.011228 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011175    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-sysconfig\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.011228 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011203    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-sys\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.011419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011230    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-kubernetes\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.011419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011256    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:22.011419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011259    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-sysconfig\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.011419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011301    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-systemd\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.011419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011323    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-run\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.011419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011333    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-kubernetes\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.011419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011347    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-sys\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.011419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011383    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-run\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.011419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011387    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-sysctl-d\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.011419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011394    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-systemd\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.011419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011416    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65bbeae6-101e-450a-bdcc-6d3c9684dad5-agent-certs\") pod \"konnectivity-agent-d44fv\" (UID: \"65bbeae6-101e-450a-bdcc-6d3c9684dad5\") " pod="kube-system/konnectivity-agent-d44fv"
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011434    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7k4t\" (UniqueName: \"kubernetes.io/projected/9224c415-9d67-4c55-9d53-4a06b3a5a44a-kube-api-access-z7k4t\") pod \"iptables-alerter-6ws95\" (UID: \"9224c415-9d67-4c55-9d53-4a06b3a5a44a\") " pod="openshift-network-operator/iptables-alerter-6ws95"
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:22.011443    2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011499    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-sysctl-conf\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:22.011507    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret podName:415a6e79-19ba-4beb-ab92-8b2b739d63a4 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:22.511486726 +0000 UTC m=+3.129068612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret") pod "global-pull-secret-syncer-xx9kc" (UID: "415a6e79-19ba-4beb-ab92-8b2b739d63a4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011522    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-sysctl-d\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011535    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/415a6e79-19ba-4beb-ab92-8b2b739d63a4-kubelet-config\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011574    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-host\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011597    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-lib-modules\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011612    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/415a6e79-19ba-4beb-ab92-8b2b739d63a4-dbus\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011638    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65bbeae6-101e-450a-bdcc-6d3c9684dad5-konnectivity-ca\") pod \"konnectivity-agent-d44fv\" (UID: \"65bbeae6-101e-450a-bdcc-6d3c9684dad5\") " pod="kube-system/konnectivity-agent-d44fv"
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011657    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-var-lib-kubelet\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011670    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23f7f8e9-e2f3-40d5-b9d5-b86176627766-tmp\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011687    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9224c415-9d67-4c55-9d53-4a06b3a5a44a-iptables-alerter-script\") pod \"iptables-alerter-6ws95\" (UID: \"9224c415-9d67-4c55-9d53-4a06b3a5a44a\") " pod="openshift-network-operator/iptables-alerter-6ws95"
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011712    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-host\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011783    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-var-lib-kubelet\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011801    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/415a6e79-19ba-4beb-ab92-8b2b739d63a4-kubelet-config\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:22.012010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011841    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/415a6e79-19ba-4beb-ab92-8b2b739d63a4-dbus\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:22.012812 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011872    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-sysctl-conf\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.012812 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.011887    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23f7f8e9-e2f3-40d5-b9d5-b86176627766-lib-modules\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.012812 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.012359    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65bbeae6-101e-450a-bdcc-6d3c9684dad5-konnectivity-ca\") pod \"konnectivity-agent-d44fv\" (UID: \"65bbeae6-101e-450a-bdcc-6d3c9684dad5\") " pod="kube-system/konnectivity-agent-d44fv"
Jul 28 20:47:22.012812 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.012382    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9224c415-9d67-4c55-9d53-4a06b3a5a44a-iptables-alerter-script\") pod \"iptables-alerter-6ws95\" (UID: \"9224c415-9d67-4c55-9d53-4a06b3a5a44a\") " pod="openshift-network-operator/iptables-alerter-6ws95"
Jul 28 20:47:22.013449 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.013400    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23f7f8e9-e2f3-40d5-b9d5-b86176627766-etc-tuned\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.014020 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.014004    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65bbeae6-101e-450a-bdcc-6d3c9684dad5-agent-certs\") pod \"konnectivity-agent-d44fv\" (UID: \"65bbeae6-101e-450a-bdcc-6d3c9684dad5\") " pod="kube-system/konnectivity-agent-d44fv"
Jul 28 20:47:22.014168 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.014150    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23f7f8e9-e2f3-40d5-b9d5-b86176627766-tmp\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.021716 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.021698    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7k4t\" (UniqueName: \"kubernetes.io/projected/9224c415-9d67-4c55-9d53-4a06b3a5a44a-kube-api-access-z7k4t\") pod \"iptables-alerter-6ws95\" (UID: \"9224c415-9d67-4c55-9d53-4a06b3a5a44a\") " pod="openshift-network-operator/iptables-alerter-6ws95"
Jul 28 20:47:22.021826 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.021806    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj2p5\" (UniqueName: \"kubernetes.io/projected/23f7f8e9-e2f3-40d5-b9d5-b86176627766-kube-api-access-fj2p5\") pod \"tuned-j9pmm\" (UID: \"23f7f8e9-e2f3-40d5-b9d5-b86176627766\") " pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.094289 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.094258    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t"
Jul 28 20:47:22.102530 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.102509    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mkp2r"
Jul 28 20:47:22.114360 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.114335    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6vhdp"
Jul 28 20:47:22.121034 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.121018    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-drhrl"
Jul 28 20:47:22.127698 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.127682    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:22.135350 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.135331    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tk54b"
Jul 28 20:47:22.142954 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.142936    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j9pmm"
Jul 28 20:47:22.155584 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.155566    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d44fv"
Jul 28 20:47:22.162093 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.162077    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6ws95"
Jul 28 20:47:22.413786 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.413701    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs\") pod \"network-metrics-daemon-9qhw8\" (UID: \"5e0b9b3f-4257-4183-90eb-1b7102f3606d\") " pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:22.413949 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:22.413820    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:22.413949 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:22.413892    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs podName:5e0b9b3f-4257-4183-90eb-1b7102f3606d nodeName:}" failed. No retries permitted until 2026-07-28 20:47:23.413874722 +0000 UTC m=+4.031456608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs") pod "network-metrics-daemon-9qhw8" (UID: "5e0b9b3f-4257-4183-90eb-1b7102f3606d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:22.514840 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.514805    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mzq\" (UniqueName: \"kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq\") pod \"network-check-target-lw7l6\" (UID: \"59627965-db98-4b3f-82ce-b9621a1eef88\") " pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:22.515029 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.514902    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:22.515029 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:22.514981    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:47:22.515029 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:22.515005    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:47:22.515029 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:22.515009    2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:22.515029 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:22.515015    2537 projected.go:196] Error preparing data for projected volume kube-api-access-m4mzq for pod openshift-network-diagnostics/network-check-target-lw7l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:22.515274 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:22.515072    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret podName:415a6e79-19ba-4beb-ab92-8b2b739d63a4 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:23.515054083 +0000 UTC m=+4.132635964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret") pod "global-pull-secret-syncer-xx9kc" (UID: "415a6e79-19ba-4beb-ab92-8b2b739d63a4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:22.515274 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:22.515091    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq podName:59627965-db98-4b3f-82ce-b9621a1eef88 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:23.51508158 +0000 UTC m=+4.132663468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4mzq" (UniqueName: "kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq") pod "network-check-target-lw7l6" (UID: "59627965-db98-4b3f-82ce-b9621a1eef88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:22.847843 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.847770    2537 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 20:42:21 +0000 UTC" deadline="2028-01-29 18:11:09.160647386 +0000 UTC"
Jul 28 20:47:22.847843 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:22.847806    2537 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13197h23m46.312845648s"
Jul 28 20:47:22.875090 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:22.875056    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0b5897_6159_4391_a86a_107312cea21e.slice/crio-00eedb50c39ee87319f983c93be48da751e822a1d8d5e63c627264fd486b8f88 WatchSource:0}: Error finding container 00eedb50c39ee87319f983c93be48da751e822a1d8d5e63c627264fd486b8f88: Status 404 returned error can't find the container with id 00eedb50c39ee87319f983c93be48da751e822a1d8d5e63c627264fd486b8f88
Jul 28 20:47:22.878132 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:22.878104    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bbeae6_101e_450a_bdcc_6d3c9684dad5.slice/crio-ab9bd12f2a6c27ccd1cb1e46336830a1143f8726e4eb56b993310787a066a22e WatchSource:0}: Error finding container ab9bd12f2a6c27ccd1cb1e46336830a1143f8726e4eb56b993310787a066a22e: Status 404 returned error can't find the container with id ab9bd12f2a6c27ccd1cb1e46336830a1143f8726e4eb56b993310787a066a22e
Jul 28 20:47:22.880366 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:22.880341    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9224c415_9d67_4c55_9d53_4a06b3a5a44a.slice/crio-a562d70f8eea9cd3bfae456b547751aefd1705dbf09e4c7a5f0fb40a197329d9 WatchSource:0}: Error finding container a562d70f8eea9cd3bfae456b547751aefd1705dbf09e4c7a5f0fb40a197329d9: Status 404 returned error can't find the container with id a562d70f8eea9cd3bfae456b547751aefd1705dbf09e4c7a5f0fb40a197329d9
Jul 28 20:47:22.881136 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:22.881071    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4a9a4e_beb8_47a6_ad56_a375b0412c2a.slice/crio-fc10c585e195864daf8253a8fbcb43fa6fb9243587d7af3eaa99e4585bf49a44 WatchSource:0}: Error finding container fc10c585e195864daf8253a8fbcb43fa6fb9243587d7af3eaa99e4585bf49a44: Status 404 returned error can't find the container with id fc10c585e195864daf8253a8fbcb43fa6fb9243587d7af3eaa99e4585bf49a44
Jul 28 20:47:22.882213 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:22.882194    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ba6b7f_59fc_4288_aef0_a479c989e6d7.slice/crio-ca7f7e0407e2cf54d29b8a591a70f14305ad57d344184833083861965ed0872e WatchSource:0}: Error finding container ca7f7e0407e2cf54d29b8a591a70f14305ad57d344184833083861965ed0872e: Status 404 returned error can't find the container with id ca7f7e0407e2cf54d29b8a591a70f14305ad57d344184833083861965ed0872e
Jul 28 20:47:22.884789 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:22.883419    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d984108_626d_4797_9164_85cc0c199bcc.slice/crio-ab62a09f5c063798ad3242b329d83f8bf14d8c1a6f888117f6f251b971b8b509 WatchSource:0}: Error finding container ab62a09f5c063798ad3242b329d83f8bf14d8c1a6f888117f6f251b971b8b509: Status 404 returned error can't find the container with id ab62a09f5c063798ad3242b329d83f8bf14d8c1a6f888117f6f251b971b8b509
Jul 28 20:47:22.884789 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:22.884254    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4421ad59_d310_4fce_b6f2_5518ede532fa.slice/crio-0ef7f456f74ea9afb4efbbff27bcd41252c5acfe92b3fe52dda60bebfd7cf012 WatchSource:0}: Error finding container 0ef7f456f74ea9afb4efbbff27bcd41252c5acfe92b3fe52dda60bebfd7cf012: Status 404 returned error can't find the container with id 0ef7f456f74ea9afb4efbbff27bcd41252c5acfe92b3fe52dda60bebfd7cf012
Jul 28 20:47:22.885920 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:22.885475    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde461c7c_9806_4aac_8afc_47e97696b0fb.slice/crio-f14386f35eebf4c8626017b556f3054f83538d54abbbb4a76a43528612392273 WatchSource:0}: Error finding container f14386f35eebf4c8626017b556f3054f83538d54abbbb4a76a43528612392273: Status 404 returned error can't find the container with id f14386f35eebf4c8626017b556f3054f83538d54abbbb4a76a43528612392273
Jul 28 20:47:22.886181 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:22.886158    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f7f8e9_e2f3_40d5_b9d5_b86176627766.slice/crio-c2c44822383e97f4cb3cc0ea92c232af32f950565b539cee655b4d098c3a22ef WatchSource:0}: Error finding container c2c44822383e97f4cb3cc0ea92c232af32f950565b539cee655b4d098c3a22ef: Status 404 returned error can't find the container with id c2c44822383e97f4cb3cc0ea92c232af32f950565b539cee655b4d098c3a22ef
Jul 28 20:47:23.014850 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.014704    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:23.014850 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:23.014966    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:23.022227 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.022205    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t" event={"ID":"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a","Type":"ContainerStarted","Data":"fc10c585e195864daf8253a8fbcb43fa6fb9243587d7af3eaa99e4585bf49a44"}
Jul 28 20:47:23.023153 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.023131    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6ws95" event={"ID":"9224c415-9d67-4c55-9d53-4a06b3a5a44a","Type":"ContainerStarted","Data":"a562d70f8eea9cd3bfae456b547751aefd1705dbf09e4c7a5f0fb40a197329d9"}
Jul 28 20:47:23.024142 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.024118    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d44fv" event={"ID":"65bbeae6-101e-450a-bdcc-6d3c9684dad5","Type":"ContainerStarted","Data":"ab9bd12f2a6c27ccd1cb1e46336830a1143f8726e4eb56b993310787a066a22e"}
Jul 28 20:47:23.025214 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.025190    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r" event={"ID":"de461c7c-9806-4aac-8afc-47e97696b0fb","Type":"ContainerStarted","Data":"f14386f35eebf4c8626017b556f3054f83538d54abbbb4a76a43528612392273"}
Jul 28 20:47:23.026197 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.026178    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mkp2r" event={"ID":"7d984108-626d-4797-9164-85cc0c199bcc","Type":"ContainerStarted","Data":"ab62a09f5c063798ad3242b329d83f8bf14d8c1a6f888117f6f251b971b8b509"}
Jul 28 20:47:23.027209 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.027188    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhrl" event={"ID":"bb0b5897-6159-4391-a86a-107312cea21e","Type":"ContainerStarted","Data":"00eedb50c39ee87319f983c93be48da751e822a1d8d5e63c627264fd486b8f88"}
Jul 28 20:47:23.029064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.029041    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-14.ec2.internal" event={"ID":"055a7a05e8fd2212ddcaf08826be7843","Type":"ContainerStarted","Data":"b14b6c23364c71ff871dfc38974c437299dc9a79ade935ede1373178e46527c0"}
Jul 28 20:47:23.030326 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.030305    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j9pmm" event={"ID":"23f7f8e9-e2f3-40d5-b9d5-b86176627766","Type":"ContainerStarted","Data":"c2c44822383e97f4cb3cc0ea92c232af32f950565b539cee655b4d098c3a22ef"}
Jul 28 20:47:23.031433 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.031412    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tk54b" event={"ID":"4421ad59-d310-4fce-b6f2-5518ede532fa","Type":"ContainerStarted","Data":"0ef7f456f74ea9afb4efbbff27bcd41252c5acfe92b3fe52dda60bebfd7cf012"}
Jul 28 20:47:23.032389 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.032371    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6vhdp" event={"ID":"19ba6b7f-59fc-4288-aef0-a479c989e6d7","Type":"ContainerStarted","Data":"ca7f7e0407e2cf54d29b8a591a70f14305ad57d344184833083861965ed0872e"}
Jul 28 20:47:23.421081 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.420358    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs\") pod \"network-metrics-daemon-9qhw8\" (UID: \"5e0b9b3f-4257-4183-90eb-1b7102f3606d\") " pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:23.421081 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:23.420554    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:23.421081 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:23.420619    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs podName:5e0b9b3f-4257-4183-90eb-1b7102f3606d nodeName:}" failed. No retries permitted until 2026-07-28 20:47:25.420598081 +0000 UTC m=+6.038179960 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs") pod "network-metrics-daemon-9qhw8" (UID: "5e0b9b3f-4257-4183-90eb-1b7102f3606d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:23.522451 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.521634    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:23.522451 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.521699    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mzq\" (UniqueName: \"kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq\") pod \"network-check-target-lw7l6\" (UID: \"59627965-db98-4b3f-82ce-b9621a1eef88\") " pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:23.522451 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:23.521874    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:47:23.522451 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:23.521892    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:47:23.522451 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:23.521905    2537 projected.go:196] Error preparing data for projected volume kube-api-access-m4mzq for pod openshift-network-diagnostics/network-check-target-lw7l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:23.522451 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:23.521959    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq podName:59627965-db98-4b3f-82ce-b9621a1eef88 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:25.521941793 +0000 UTC m=+6.139523690 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4mzq" (UniqueName: "kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq") pod "network-check-target-lw7l6" (UID: "59627965-db98-4b3f-82ce-b9621a1eef88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:23.522451 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:23.522328    2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:23.522451 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:23.522375    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret podName:415a6e79-19ba-4beb-ab92-8b2b739d63a4 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:25.52236135 +0000 UTC m=+6.139943238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret") pod "global-pull-secret-syncer-xx9kc" (UID: "415a6e79-19ba-4beb-ab92-8b2b739d63a4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:23.848580 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.848473    2537 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 20:42:21 +0000 UTC" deadline="2028-02-24 12:43:49.184776367 +0000 UTC"
Jul 28 20:47:23.848580 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:23.848536    2537 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13815h56m25.336244586s"
Jul 28 20:47:24.016007 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:24.015003    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:24.016007 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:24.015153    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:24.016007 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:24.015533    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:24.016007 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:24.015626    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:24.045517 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:24.045481    2537 generic.go:356] "Generic (PLEG): container finished" podID="95e38ebd473179562ecd1ee1c342a23a" containerID="ffdcab24f34a3411256b62e0cb388748f8d6474ddded06d3d0ac33d7b55cd5eb" exitCode=0
Jul 28 20:47:24.046432 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:24.046403    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal" event={"ID":"95e38ebd473179562ecd1ee1c342a23a","Type":"ContainerDied","Data":"ffdcab24f34a3411256b62e0cb388748f8d6474ddded06d3d0ac33d7b55cd5eb"}
Jul 28 20:47:24.065878 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:24.065799    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-14.ec2.internal" podStartSLOduration=3.065779296 podStartE2EDuration="3.065779296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:23.05058435 +0000 UTC m=+3.668166254" watchObservedRunningTime="2026-07-28 20:47:24.065779296 +0000 UTC m=+4.683361199"
Jul 28 20:47:25.014904 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:25.014848    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:25.015345 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:25.015007    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:25.068492 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:25.068414    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal" event={"ID":"95e38ebd473179562ecd1ee1c342a23a","Type":"ContainerStarted","Data":"9977ae5048ab0ef9935a0b3a76a98ab810711d98548aaa76a74248158762a779"}
Jul 28 20:47:25.438832 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:25.438790    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs\") pod \"network-metrics-daemon-9qhw8\" (UID: \"5e0b9b3f-4257-4183-90eb-1b7102f3606d\") " pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:25.439029 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:25.438947    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:25.439029 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:25.439013    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs podName:5e0b9b3f-4257-4183-90eb-1b7102f3606d nodeName:}" failed. No retries permitted until 2026-07-28 20:47:29.438993971 +0000 UTC m=+10.056575874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs") pod "network-metrics-daemon-9qhw8" (UID: "5e0b9b3f-4257-4183-90eb-1b7102f3606d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:25.540659 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:25.539785    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:25.540659 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:25.539881    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mzq\" (UniqueName: \"kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq\") pod \"network-check-target-lw7l6\" (UID: \"59627965-db98-4b3f-82ce-b9621a1eef88\") " pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:25.540659 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:25.540058    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:47:25.540659 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:25.540077    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:47:25.540659 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:25.540090    2537 projected.go:196] Error preparing data for projected volume kube-api-access-m4mzq for pod openshift-network-diagnostics/network-check-target-lw7l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:25.540659 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:25.540145    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq podName:59627965-db98-4b3f-82ce-b9621a1eef88 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:29.540127348 +0000 UTC m=+10.157709240 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4mzq" (UniqueName: "kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq") pod "network-check-target-lw7l6" (UID: "59627965-db98-4b3f-82ce-b9621a1eef88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:25.540659 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:25.540527    2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:25.540659 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:25.540575    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret podName:415a6e79-19ba-4beb-ab92-8b2b739d63a4 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:29.540560611 +0000 UTC m=+10.158142495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret") pod "global-pull-secret-syncer-xx9kc" (UID: "415a6e79-19ba-4beb-ab92-8b2b739d63a4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:26.019757 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:26.019678    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:26.020280 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:26.019802    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:26.020371 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:26.020291    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:26.020427 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:26.020398    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:27.015867 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:27.015366    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:27.015867 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:27.015498    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:28.015272 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:28.014815    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:28.015272 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:28.014991    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:28.015272 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:28.015034    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:28.015272 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:28.015114    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:29.014823 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:29.014750    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:29.015007 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:29.014909    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:29.471536 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:29.471366    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs\") pod \"network-metrics-daemon-9qhw8\" (UID: \"5e0b9b3f-4257-4183-90eb-1b7102f3606d\") " pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:29.472052 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:29.471567    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:29.472052 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:29.471634    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs podName:5e0b9b3f-4257-4183-90eb-1b7102f3606d nodeName:}" failed. No retries permitted until 2026-07-28 20:47:37.471613776 +0000 UTC m=+18.089195669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs") pod "network-metrics-daemon-9qhw8" (UID: "5e0b9b3f-4257-4183-90eb-1b7102f3606d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:29.572806 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:29.572767    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:29.573000 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:29.572837    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mzq\" (UniqueName: \"kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq\") pod \"network-check-target-lw7l6\" (UID: \"59627965-db98-4b3f-82ce-b9621a1eef88\") " pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:29.573000 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:29.572904    2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:29.573000 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:29.572976    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret podName:415a6e79-19ba-4beb-ab92-8b2b739d63a4 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:37.57295648 +0000 UTC m=+18.190538362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret") pod "global-pull-secret-syncer-xx9kc" (UID: "415a6e79-19ba-4beb-ab92-8b2b739d63a4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:29.573193 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:29.573010    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:47:29.573193 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:29.573029    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:47:29.573193 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:29.573044    2537 projected.go:196] Error preparing data for projected volume kube-api-access-m4mzq for pod openshift-network-diagnostics/network-check-target-lw7l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:29.573193 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:29.573091    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq podName:59627965-db98-4b3f-82ce-b9621a1eef88 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:37.573076596 +0000 UTC m=+18.190658495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4mzq" (UniqueName: "kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq") pod "network-check-target-lw7l6" (UID: "59627965-db98-4b3f-82ce-b9621a1eef88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:30.016433 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:30.016227    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:30.016433 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:30.016363    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:30.016433 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:30.016408    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:30.016790 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:30.016475    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:31.015102 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:31.015023    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:31.015540 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:31.015139    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:32.015227 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:32.015192    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:32.015710 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:32.015192    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:32.015710 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:32.015344    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:32.015710 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:32.015387    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:33.015002 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:33.014948    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:33.015171 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:33.015112    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:34.014803 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:34.014766    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:34.015243 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:34.014906    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:34.015243 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:34.014959    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:34.015243 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:34.015079    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:35.014540 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:35.014505    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:35.014710 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:35.014628    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:36.014893 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:36.014845    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:36.015342 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:36.014874    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:36.015342 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:36.014986    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:36.015342 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:36.015072    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:37.014866 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:37.014831    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:37.015089 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:37.014943    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:37.532439 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:37.532389    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs\") pod \"network-metrics-daemon-9qhw8\" (UID: \"5e0b9b3f-4257-4183-90eb-1b7102f3606d\") " pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:37.532685 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:37.532554    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:37.532685 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:37.532615    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs podName:5e0b9b3f-4257-4183-90eb-1b7102f3606d nodeName:}" failed. No retries permitted until 2026-07-28 20:47:53.532595199 +0000 UTC m=+34.150177080 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs") pod "network-metrics-daemon-9qhw8" (UID: "5e0b9b3f-4257-4183-90eb-1b7102f3606d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:37.633608 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:37.633571    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:37.633776 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:37.633628    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mzq\" (UniqueName: \"kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq\") pod \"network-check-target-lw7l6\" (UID: \"59627965-db98-4b3f-82ce-b9621a1eef88\") " pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:37.633776 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:37.633723    2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:37.633776 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:37.633744    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:47:37.633776 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:37.633759    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:47:37.633776 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:37.633768    2537 projected.go:196] Error preparing data for projected volume kube-api-access-m4mzq for pod openshift-network-diagnostics/network-check-target-lw7l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:37.633996 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:37.633793    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret podName:415a6e79-19ba-4beb-ab92-8b2b739d63a4 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:53.633770589 +0000 UTC m=+34.251352472 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret") pod "global-pull-secret-syncer-xx9kc" (UID: "415a6e79-19ba-4beb-ab92-8b2b739d63a4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:37.633996 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:37.633811    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq podName:59627965-db98-4b3f-82ce-b9621a1eef88 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:53.633800787 +0000 UTC m=+34.251382668 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4mzq" (UniqueName: "kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq") pod "network-check-target-lw7l6" (UID: "59627965-db98-4b3f-82ce-b9621a1eef88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:38.015408 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:38.015329    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:38.015846 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:38.015454    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:38.015846 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:38.015512    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:38.015846 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:38.015610    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:39.015027 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:39.014994    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:39.015222 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:39.015129    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:40.018167 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:40.018135    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:40.018622 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:40.018263    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:40.018622 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:40.018341    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:40.020435 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:40.020396    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:41.014973 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:41.014919    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:41.015189 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:41.015060    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:42.015535 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.015306    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:42.016071 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.015370    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:42.016071 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:42.015670    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:42.016071 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:42.015745    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:42.103355 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.103329    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j9pmm" event={"ID":"23f7f8e9-e2f3-40d5-b9d5-b86176627766","Type":"ContainerStarted","Data":"09ea0e235a3879ae76ab4cbeb1c9471d93370149a5b6fd4e41ffadeb457b4905"}
Jul 28 20:47:42.104873 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.104831    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tk54b" event={"ID":"4421ad59-d310-4fce-b6f2-5518ede532fa","Type":"ContainerStarted","Data":"6a651fb1b8836028faa3df2e01c35b9653b53e2019c282bcf79e3b6bf723e819"}
Jul 28 20:47:42.106228 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.106196    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6vhdp" event={"ID":"19ba6b7f-59fc-4288-aef0-a479c989e6d7","Type":"ContainerStarted","Data":"72f8d14e9f8a9403a9ec6a43b93d55ae38b7d2dbf1e5d9703f6fadda8ad112be"}
Jul 28 20:47:42.107567 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.107547    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t" event={"ID":"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a","Type":"ContainerStarted","Data":"bcf0b1e80e584b620f8b505a39bd4ec7c6b251253d8965040c2e8fb4cedaa48c"}
Jul 28 20:47:42.108925 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.108904    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d44fv" event={"ID":"65bbeae6-101e-450a-bdcc-6d3c9684dad5","Type":"ContainerStarted","Data":"afc52905459eb644fe6713331acda2c1e1c88dda0638b6d8ff69aa6974e9a527"}
Jul 28 20:47:42.111216 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.111198    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 20:47:42.111610 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.111586    2537 generic.go:356] "Generic (PLEG): container finished" podID="de461c7c-9806-4aac-8afc-47e97696b0fb" containerID="255753973cc4efe9f25b7ddc3b93d8c375324617e3a5728ac569c45a39ca60eb" exitCode=1
Jul 28 20:47:42.111686 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.111656    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r" event={"ID":"de461c7c-9806-4aac-8afc-47e97696b0fb","Type":"ContainerStarted","Data":"7b4a559cf5221d86d326f418044216b62d5e4783521ad57d9ade23dfada1dab0"}
Jul 28 20:47:42.111735 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.111728    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r" event={"ID":"de461c7c-9806-4aac-8afc-47e97696b0fb","Type":"ContainerStarted","Data":"2919a598bfc15e67bb70bf3c5b7ec0acdab9637ba1a6b63bfcb3f9ac3d656b52"}
Jul 28 20:47:42.111789 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.111775    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r" event={"ID":"de461c7c-9806-4aac-8afc-47e97696b0fb","Type":"ContainerStarted","Data":"1de0cdadb388aef59258f1d4526660f02dd36c1adf0fa4e2c853783337152c2f"}
Jul 28 20:47:42.111841 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.111828    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r" event={"ID":"de461c7c-9806-4aac-8afc-47e97696b0fb","Type":"ContainerDied","Data":"255753973cc4efe9f25b7ddc3b93d8c375324617e3a5728ac569c45a39ca60eb"}
Jul 28 20:47:42.111922 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.111908    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r" event={"ID":"de461c7c-9806-4aac-8afc-47e97696b0fb","Type":"ContainerStarted","Data":"c94b71b8c87eaad84551b404ffa5b8b85f252389c1fabbf5027679c3149242cc"}
Jul 28 20:47:42.113449 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.113426    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mkp2r" event={"ID":"7d984108-626d-4797-9164-85cc0c199bcc","Type":"ContainerStarted","Data":"1bc62e90b0daf8e3524b6d47268e52ae41baeff478f7744a9f7e18cbe681f86d"}
Jul 28 20:47:42.114883 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.114838    2537 generic.go:356] "Generic (PLEG): container finished" podID="bb0b5897-6159-4391-a86a-107312cea21e" containerID="921502772b4ffbc17fcff61eca8da38afb35aad44606daa6fc9748a823bd2eb1" exitCode=0
Jul 28 20:47:42.114961 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.114882    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhrl" event={"ID":"bb0b5897-6159-4391-a86a-107312cea21e","Type":"ContainerDied","Data":"921502772b4ffbc17fcff61eca8da38afb35aad44606daa6fc9748a823bd2eb1"}
Jul 28 20:47:42.123642 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.123609    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j9pmm" podStartSLOduration=3.857082308 podStartE2EDuration="22.123598685s" totalImagesPullingTime="18.266516377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:42.123165484 +0000 UTC m=+22.740747385" watchObservedRunningTime="2026-07-28 20:47:42.123598685 +0000 UTC m=+22.741180587"
Jul 28 20:47:42.123763 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.123736    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-14.ec2.internal" podStartSLOduration=21.123727087 podStartE2EDuration="21.123727087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:25.089971342 +0000 UTC m=+5.707553239" watchObservedRunningTime="2026-07-28 20:47:42.123727087 +0000 UTC m=+22.741309184"
Jul 28 20:47:42.141247 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.141213    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d44fv" podStartSLOduration=11.704574828 podStartE2EDuration="22.141200564s" totalImagesPullingTime="10.436625736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:42.140555866 +0000 UTC m=+22.758137769" watchObservedRunningTime="2026-07-28 20:47:42.141200564 +0000 UTC m=+22.758782472"
Jul 28 20:47:42.162276 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.162232    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mkp2r" podStartSLOduration=3.8625340489999997 podStartE2EDuration="22.162216931s" totalImagesPullingTime="18.299682882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:42.161994213 +0000 UTC m=+22.779576118" watchObservedRunningTime="2026-07-28 20:47:42.162216931 +0000 UTC m=+22.779798845"
Jul 28 20:47:42.203569 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.203524    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6vhdp" podStartSLOduration=3.915577719 podStartE2EDuration="22.203505542s" totalImagesPullingTime="18.287927823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:42.178506315 +0000 UTC m=+22.796088217" watchObservedRunningTime="2026-07-28 20:47:42.203505542 +0000 UTC m=+22.821087458"
Jul 28 20:47:42.219815 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.219770    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tk54b" podStartSLOduration=3.956569111 podStartE2EDuration="22.219754194s" totalImagesPullingTime="18.263185083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:42.219440698 +0000 UTC m=+22.837022601" watchObservedRunningTime="2026-07-28 20:47:42.219754194 +0000 UTC m=+22.837336097"
Jul 28 20:47:42.636593 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.636569    2537 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 20:47:42.997077 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.996899    2537 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T20:47:42.636589445Z","UUID":"bbd6cfb2-cd93-4a55-bfd1-1364cd27a40a","Handler":null,"Name":"","Endpoint":""}
Jul 28 20:47:42.998496 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.998474    2537 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 20:47:42.998632 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:42.998515    2537 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 20:47:43.014646 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:43.014624    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:43.014789 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:43.014724    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:43.119518 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:43.119482    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t" event={"ID":"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a","Type":"ContainerStarted","Data":"9d9e1ce7875c6197419cb9cd94de42321eff266f12f1e2172204d441e46eb31d"}
Jul 28 20:47:43.121891 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:43.121496    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6ws95" event={"ID":"9224c415-9d67-4c55-9d53-4a06b3a5a44a","Type":"ContainerStarted","Data":"542a22cd7ea5bce710e49dba5860ed0a39e11652b116a74d744cf3352088f211"}
Jul 28 20:47:43.127412 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:43.127392    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 20:47:43.127892 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:43.127846    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r" event={"ID":"de461c7c-9806-4aac-8afc-47e97696b0fb","Type":"ContainerStarted","Data":"bdf29622588c18ef7759ec2d4d66bf8173bbfdc377a4b4af2b23a59ed86ab36f"}
Jul 28 20:47:44.015381 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:44.015191    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:44.015537 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:44.015242    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:44.015600 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:44.015526    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:44.015694 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:44.015666    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:44.132821 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:44.132772    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t" event={"ID":"8b4a9a4e-beb8-47a6-ad56-a375b0412c2a","Type":"ContainerStarted","Data":"880f8bf9693d4946b38863c7b125681d121fc4d13dd8136f3555c563593802b3"}
Jul 28 20:47:44.154981 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:44.154910    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6ws95" podStartSLOduration=5.8647183080000005 podStartE2EDuration="24.154889684s" totalImagesPullingTime="18.290171376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:43.140112057 +0000 UTC m=+23.757693958" watchObservedRunningTime="2026-07-28 20:47:44.154889684 +0000 UTC m=+24.772471587"
Jul 28 20:47:44.626573 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:44.626497    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d44fv"
Jul 28 20:47:44.627247 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:44.627224    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d44fv"
Jul 28 20:47:44.647566 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:44.647519    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l886t" podStartSLOduration=3.950317188 podStartE2EDuration="24.64750022s" totalImagesPullingTime="20.697183032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:44.154743027 +0000 UTC m=+24.772324940" watchObservedRunningTime="2026-07-28 20:47:44.64750022 +0000 UTC m=+25.265082123"
Jul 28 20:47:45.015547 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:45.015473    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:45.015737 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:45.015591    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:45.139073 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:45.138998    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 20:47:45.140640 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:45.140370    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r" event={"ID":"de461c7c-9806-4aac-8afc-47e97696b0fb","Type":"ContainerStarted","Data":"21eae06c712e7546f9be7c1bdf644ad0f4f31aa3a7344a3eda04ecd583c16dda"}
Jul 28 20:47:45.141144 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:45.140973    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d44fv"
Jul 28 20:47:45.141516 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:45.141488    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d44fv"
Jul 28 20:47:46.015601 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:46.015571    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:46.015775 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:46.015674    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:46.015775 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:46.015769    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:46.015978 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:46.015871    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:47.015009 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:47.014703    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:47.015614 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:47.015022    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:47.146719 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:47.146694    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 20:47:47.147103 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:47.147078    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r" event={"ID":"de461c7c-9806-4aac-8afc-47e97696b0fb","Type":"ContainerStarted","Data":"281ee187dd9f5c6a429da0761ad5411ee7c5f43d8d9a9190f2e8b149b3faf0d0"}
Jul 28 20:47:47.147471 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:47.147386    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:47.147585 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:47.147533    2537 scope.go:117] "RemoveContainer" containerID="255753973cc4efe9f25b7ddc3b93d8c375324617e3a5728ac569c45a39ca60eb"
Jul 28 20:47:47.148906 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:47.148882    2537 generic.go:356] "Generic (PLEG): container finished" podID="bb0b5897-6159-4391-a86a-107312cea21e" containerID="c8f6e2e7f20110de6df337411835ada814f4cf31975483d07fff33de4f2c6211" exitCode=0
Jul 28 20:47:47.149026 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:47.148965    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhrl" event={"ID":"bb0b5897-6159-4391-a86a-107312cea21e","Type":"ContainerDied","Data":"c8f6e2e7f20110de6df337411835ada814f4cf31975483d07fff33de4f2c6211"}
Jul 28 20:47:47.164129 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:47.164113    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:48.014965 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:48.014931    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:48.014965 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:48.014962    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:48.015416 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:48.015031    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:48.015416 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:48.015175    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:48.156702 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:48.156672    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 20:47:48.157064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:48.157037    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r" event={"ID":"de461c7c-9806-4aac-8afc-47e97696b0fb","Type":"ContainerStarted","Data":"6b0d383024841672af37e884a50fe4470ecd88b7a3c10ea77c01e9ce02d22939"}
Jul 28 20:47:48.157322 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:48.157298    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:48.157437 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:48.157371    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:48.171330 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:48.171308    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:47:48.225282 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:48.225240    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r" podStartSLOduration=9.485691871 podStartE2EDuration="28.225225052s" totalImagesPullingTime="18.739533181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:48.193104056 +0000 UTC m=+28.810685961" watchObservedRunningTime="2026-07-28 20:47:48.225225052 +0000 UTC m=+28.842806954"
Jul 28 20:47:48.724343 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:48.724145    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lw7l6"]
Jul 28 20:47:48.724536 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:48.724429    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:48.724536 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:48.724521    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:48.727371 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:48.727348    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xx9kc"]
Jul 28 20:47:48.727459 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:48.727431    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:48.727514 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:48.727497    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:48.729643 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:48.729621    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9qhw8"]
Jul 28 20:47:48.729767 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:48.729754    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:48.729880 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:48.729845    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:49.160725 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:49.160692    2537 generic.go:356] "Generic (PLEG): container finished" podID="bb0b5897-6159-4391-a86a-107312cea21e" containerID="f8af2411c58b09efd0f94444b0a2df60b16baa9cf3fdd39e363a1a41eb702130" exitCode=0
Jul 28 20:47:49.161122 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:49.160776    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhrl" event={"ID":"bb0b5897-6159-4391-a86a-107312cea21e","Type":"ContainerDied","Data":"f8af2411c58b09efd0f94444b0a2df60b16baa9cf3fdd39e363a1a41eb702130"}
Jul 28 20:47:50.019937 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:50.019911    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:50.020090 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:50.019996    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:50.020090 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:50.020005    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:50.020090 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:50.020022    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:50.020194 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:50.020101    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:50.020194 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:50.020168    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:51.167242 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:51.167151    2537 generic.go:356] "Generic (PLEG): container finished" podID="bb0b5897-6159-4391-a86a-107312cea21e" containerID="6fb0c24d09058252f6b2fd7ae7f8a895f0e8c96fd0b9bdc467990d69164e7c25" exitCode=0
Jul 28 20:47:51.167242 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:51.167187    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhrl" event={"ID":"bb0b5897-6159-4391-a86a-107312cea21e","Type":"ContainerDied","Data":"6fb0c24d09058252f6b2fd7ae7f8a895f0e8c96fd0b9bdc467990d69164e7c25"}
Jul 28 20:47:52.018493 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:52.018462    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:52.018721 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:52.018521    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:52.018721 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:52.018540    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:52.018721 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:52.018633    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:52.018939 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:52.018711    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:52.018939 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:52.018787    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:53.555787 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:53.555755    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs\") pod \"network-metrics-daemon-9qhw8\" (UID: \"5e0b9b3f-4257-4183-90eb-1b7102f3606d\") " pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:53.556261 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:53.555938    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:53.556261 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:53.556020    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs podName:5e0b9b3f-4257-4183-90eb-1b7102f3606d nodeName:}" failed. No retries permitted until 2026-07-28 20:48:25.555999134 +0000 UTC m=+66.173581018 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs") pod "network-metrics-daemon-9qhw8" (UID: "5e0b9b3f-4257-4183-90eb-1b7102f3606d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:53.656850 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:53.656818    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:53.657042 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:53.656882    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mzq\" (UniqueName: \"kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq\") pod \"network-check-target-lw7l6\" (UID: \"59627965-db98-4b3f-82ce-b9621a1eef88\") " pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:53.657042 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:53.656989    2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:53.657042 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:53.657028    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:47:53.657042 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:53.657044    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:47:53.657256 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:53.657054    2537 projected.go:196] Error preparing data for projected volume kube-api-access-m4mzq for pod openshift-network-diagnostics/network-check-target-lw7l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:53.657256 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:53.657046    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret podName:415a6e79-19ba-4beb-ab92-8b2b739d63a4 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:25.657033073 +0000 UTC m=+66.274614958 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret") pod "global-pull-secret-syncer-xx9kc" (UID: "415a6e79-19ba-4beb-ab92-8b2b739d63a4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:53.657256 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:53.657102    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq podName:59627965-db98-4b3f-82ce-b9621a1eef88 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:25.657086989 +0000 UTC m=+66.274668870 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4mzq" (UniqueName: "kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq") pod "network-check-target-lw7l6" (UID: "59627965-db98-4b3f-82ce-b9621a1eef88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:54.014924 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:54.014796    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:54.014924 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:54.014796    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:54.015128 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:54.014954    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:54.015128 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:54.014957    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qhw8" podUID="5e0b9b3f-4257-4183-90eb-1b7102f3606d"
Jul 28 20:47:54.015128 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:54.015061    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw7l6" podUID="59627965-db98-4b3f-82ce-b9621a1eef88"
Jul 28 20:47:54.015249 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:54.015146    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xx9kc" podUID="415a6e79-19ba-4beb-ab92-8b2b739d63a4"
Jul 28 20:47:54.865912 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:54.865692    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:54.951573 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:54.951541    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:55.166091 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.166073    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-14.ec2.internal" event="NodeReady"
Jul 28 20:47:55.166213 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.166190    2537 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 20:47:55.234552 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.234517    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r2g2c"]
Jul 28 20:47:55.237760 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.237738    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mrnqh"]
Jul 28 20:47:55.237908 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.237890    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r2g2c"
Jul 28 20:47:55.240840 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.240818    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:47:55.241014 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.240963    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 20:47:55.241104 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.241068    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 20:47:55.241162 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.241110    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-m4gqb\""
Jul 28 20:47:55.241249 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.241227    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 20:47:55.243769 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.243749    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 20:47:55.244045 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.244030    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ghnpw\""
Jul 28 20:47:55.244439 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.244422    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 20:47:55.248695 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.248674    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mrnqh"]
Jul 28 20:47:55.256226 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.254608    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r2g2c"]
Jul 28 20:47:55.367667 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.367584    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8083d69e-72cb-48d1-91bb-286ee0b4141d-cert\") pod \"ingress-canary-r2g2c\" (UID: \"8083d69e-72cb-48d1-91bb-286ee0b4141d\") " pod="openshift-ingress-canary/ingress-canary-r2g2c"
Jul 28 20:47:55.367667 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.367614    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:47:55.367667 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.367647    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b960510-a109-42a8-99d8-dad53a778237-config-volume\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:47:55.367902 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.367735    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b960510-a109-42a8-99d8-dad53a778237-tmp-dir\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:47:55.367902 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.367808    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpqbd\" (UniqueName: \"kubernetes.io/projected/8083d69e-72cb-48d1-91bb-286ee0b4141d-kube-api-access-cpqbd\") pod \"ingress-canary-r2g2c\" (UID: \"8083d69e-72cb-48d1-91bb-286ee0b4141d\") " pod="openshift-ingress-canary/ingress-canary-r2g2c"
Jul 28 20:47:55.367902 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.367845    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z52kl\" (UniqueName: \"kubernetes.io/projected/0b960510-a109-42a8-99d8-dad53a778237-kube-api-access-z52kl\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:47:55.441065 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.441039    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:55.468248 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.468228    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:47:55.468317 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.468270    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b960510-a109-42a8-99d8-dad53a778237-config-volume\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:47:55.468317 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.468294    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b960510-a109-42a8-99d8-dad53a778237-tmp-dir\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:47:55.468389 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.468323    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpqbd\" (UniqueName: \"kubernetes.io/projected/8083d69e-72cb-48d1-91bb-286ee0b4141d-kube-api-access-cpqbd\") pod \"ingress-canary-r2g2c\" (UID: \"8083d69e-72cb-48d1-91bb-286ee0b4141d\") " pod="openshift-ingress-canary/ingress-canary-r2g2c"
Jul 28 20:47:55.468389 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.468342    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z52kl\" (UniqueName: \"kubernetes.io/projected/0b960510-a109-42a8-99d8-dad53a778237-kube-api-access-z52kl\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:47:55.468389 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.468363    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8083d69e-72cb-48d1-91bb-286ee0b4141d-cert\") pod \"ingress-canary-r2g2c\" (UID: \"8083d69e-72cb-48d1-91bb-286ee0b4141d\") " pod="openshift-ingress-canary/ingress-canary-r2g2c"
Jul 28 20:47:55.468389 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:55.468372    2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:47:55.468570 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:55.468455    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls podName:0b960510-a109-42a8-99d8-dad53a778237 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:55.96843538 +0000 UTC m=+36.586017261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls") pod "dns-default-mrnqh" (UID: "0b960510-a109-42a8-99d8-dad53a778237") : secret "dns-default-metrics-tls" not found
Jul 28 20:47:55.468695 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.468676    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b960510-a109-42a8-99d8-dad53a778237-tmp-dir\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:47:55.468785 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.468769    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b960510-a109-42a8-99d8-dad53a778237-config-volume\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:47:55.472371 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.472355    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8083d69e-72cb-48d1-91bb-286ee0b4141d-cert\") pod \"ingress-canary-r2g2c\" (UID: \"8083d69e-72cb-48d1-91bb-286ee0b4141d\") " pod="openshift-ingress-canary/ingress-canary-r2g2c"
Jul 28 20:47:55.484774 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.484750    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpqbd\" (UniqueName: \"kubernetes.io/projected/8083d69e-72cb-48d1-91bb-286ee0b4141d-kube-api-access-cpqbd\") pod \"ingress-canary-r2g2c\" (UID: \"8083d69e-72cb-48d1-91bb-286ee0b4141d\") " pod="openshift-ingress-canary/ingress-canary-r2g2c"
Jul 28 20:47:55.485986 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.485968    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z52kl\" (UniqueName: \"kubernetes.io/projected/0b960510-a109-42a8-99d8-dad53a778237-kube-api-access-z52kl\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:47:55.525377 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.525352    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:55.549286 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.549267    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r2g2c"
Jul 28 20:47:55.688162 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.688116    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r2g2c"]
Jul 28 20:47:55.691950 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:47:55.691928    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8083d69e_72cb_48d1_91bb_286ee0b4141d.slice/crio-e384ffb3c605c9e7f0e1de3a02902563131cf98cc28b3171c4db32c90573c660 WatchSource:0}: Error finding container e384ffb3c605c9e7f0e1de3a02902563131cf98cc28b3171c4db32c90573c660: Status 404 returned error can't find the container with id e384ffb3c605c9e7f0e1de3a02902563131cf98cc28b3171c4db32c90573c660
Jul 28 20:47:55.971634 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:55.971545    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:47:55.972081 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:55.971674    2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:47:55.972081 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:55.971755    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls podName:0b960510-a109-42a8-99d8-dad53a778237 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:56.971735097 +0000 UTC m=+37.589316988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls") pod "dns-default-mrnqh" (UID: "0b960510-a109-42a8-99d8-dad53a778237") : secret "dns-default-metrics-tls" not found
Jul 28 20:47:56.018301 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:56.018272    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:47:56.018413 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:56.018388    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:47:56.018483 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:56.018454    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:47:56.023422 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:56.023403    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 20:47:56.023520 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:56.023504    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j52pw\""
Jul 28 20:47:56.024302 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:56.024283    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gvjsj\""
Jul 28 20:47:56.024302 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:56.024297    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 20:47:56.024491 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:56.024467    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 20:47:56.024491 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:56.024480    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 20:47:56.185748 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:56.185717    2537 generic.go:356] "Generic (PLEG): container finished" podID="bb0b5897-6159-4391-a86a-107312cea21e" containerID="e60e793b44b8b9258382a51b884140f832e776e33346262008af7b5deafa21eb" exitCode=0
Jul 28 20:47:56.185941 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:56.185805    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhrl" event={"ID":"bb0b5897-6159-4391-a86a-107312cea21e","Type":"ContainerDied","Data":"e60e793b44b8b9258382a51b884140f832e776e33346262008af7b5deafa21eb"}
Jul 28 20:47:56.187379 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:56.186905    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r2g2c" event={"ID":"8083d69e-72cb-48d1-91bb-286ee0b4141d","Type":"ContainerStarted","Data":"e384ffb3c605c9e7f0e1de3a02902563131cf98cc28b3171c4db32c90573c660"}
Jul 28 20:47:56.978414 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:56.978228    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:47:56.978848 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:56.978415    2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:47:56.978848 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:56.978489    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls podName:0b960510-a109-42a8-99d8-dad53a778237 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:58.978469703 +0000 UTC m=+39.596051596 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls") pod "dns-default-mrnqh" (UID: "0b960510-a109-42a8-99d8-dad53a778237") : secret "dns-default-metrics-tls" not found
Jul 28 20:47:57.193210 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:57.193171    2537 generic.go:356] "Generic (PLEG): container finished" podID="bb0b5897-6159-4391-a86a-107312cea21e" containerID="42bd5b7d36bf8e8a8bc053c0f905ec3e50cc0b7cf3890b798c05d99517f84ad8" exitCode=0
Jul 28 20:47:57.193369 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:57.193245    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhrl" event={"ID":"bb0b5897-6159-4391-a86a-107312cea21e","Type":"ContainerDied","Data":"42bd5b7d36bf8e8a8bc053c0f905ec3e50cc0b7cf3890b798c05d99517f84ad8"}
Jul 28 20:47:58.199832 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:58.199754    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhrl" event={"ID":"bb0b5897-6159-4391-a86a-107312cea21e","Type":"ContainerStarted","Data":"86b411b02791c447077d6c68f7619f6576367fdc38b1e70e35a9f26a22374d21"}
Jul 28 20:47:58.395561 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:58.395532    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:58.464145 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:58.464066    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:58.527351 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:58.527317    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:58.600116 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:58.600084    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:58.698950 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:58.698919    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:58.994384 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:58.994346    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:47:58.994521 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:58.994486    2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:47:58.994556 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:47:58.994542    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls podName:0b960510-a109-42a8-99d8-dad53a778237 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:02.994525219 +0000 UTC m=+43.612107104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls") pod "dns-default-mrnqh" (UID: "0b960510-a109-42a8-99d8-dad53a778237") : secret "dns-default-metrics-tls" not found
Jul 28 20:47:59.203344 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:59.203309    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r2g2c" event={"ID":"8083d69e-72cb-48d1-91bb-286ee0b4141d","Type":"ContainerStarted","Data":"7301b4db699ea1daee0058844591d198272f7a5e17887ad0e558aca890c6bc4d"}
Jul 28 20:47:59.221558 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:59.221512    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r2g2c" podStartSLOduration=1.680983093 podStartE2EDuration="4.221498738s" totalImagesPullingTime="2.540515645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:59.220230026 +0000 UTC m=+39.837811928" watchObservedRunningTime="2026-07-28 20:47:59.221498738 +0000 UTC m=+39.839080640"
Jul 28 20:47:59.221689 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:47:59.221596    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-drhrl" podStartSLOduration=10.862924778 podStartE2EDuration="39.221591028s" totalImagesPullingTime="28.35866625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:58.252007484 +0000 UTC m=+38.869589386" watchObservedRunningTime="2026-07-28 20:47:59.221591028 +0000 UTC m=+39.839172930"
Jul 28 20:48:03.021087 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:03.021041    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:48:03.021500 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:03.021194    2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:48:03.021500 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:03.021258    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls podName:0b960510-a109-42a8-99d8-dad53a778237 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:11.021242847 +0000 UTC m=+51.638824727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls") pod "dns-default-mrnqh" (UID: "0b960510-a109-42a8-99d8-dad53a778237") : secret "dns-default-metrics-tls" not found
Jul 28 20:48:03.699410 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:03.699362    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:03.763668 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:03.763641    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:04.307386 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:04.307351    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:04.361176 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:04.361148    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:11.075377 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:11.075344    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:48:11.075752 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:11.075469    2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:48:11.075752 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:11.075519    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls podName:0b960510-a109-42a8-99d8-dad53a778237 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:27.07550495 +0000 UTC m=+67.693086831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls") pod "dns-default-mrnqh" (UID: "0b960510-a109-42a8-99d8-dad53a778237") : secret "dns-default-metrics-tls" not found
Jul 28 20:48:15.824493 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:15.824451    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:15.920996 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:15.920965    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:20.179204 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.179171    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldz9r"
Jul 28 20:48:20.341629 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.341596    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:20.410809 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.410769    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:20.828561 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.828529    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5p5wl"]
Jul 28 20:48:20.831801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.831784    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5p5wl"
Jul 28 20:48:20.834674 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.834657    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ctfx2\""
Jul 28 20:48:20.835797 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.835782    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 20:48:20.835909 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.835800    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 20:48:20.842629 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.842609    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5p5wl"]
Jul 28 20:48:20.844094 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.844071    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz6vv\" (UniqueName: \"kubernetes.io/projected/41fbe0e3-2435-4381-940e-508116343854-kube-api-access-gz6vv\") pod \"volume-data-source-validator-549c6d54cf-5p5wl\" (UID: \"41fbe0e3-2435-4381-940e-508116343854\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5p5wl"
Jul 28 20:48:20.933040 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.933004    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"]
Jul 28 20:48:20.935225 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.935210    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"
Jul 28 20:48:20.937798 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.937775    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 28 20:48:20.938168 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.938151    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 28 20:48:20.938251 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.938153    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pjd25\""
Jul 28 20:48:20.938747 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.938725    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 28 20:48:20.945182 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.945160    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lblk5\" (UID: \"1aeaf8a7-0733-428a-81d7-6f57520e2936\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"
Jul 28 20:48:20.945280 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.945216    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz6vv\" (UniqueName: \"kubernetes.io/projected/41fbe0e3-2435-4381-940e-508116343854-kube-api-access-gz6vv\") pod \"volume-data-source-validator-549c6d54cf-5p5wl\" (UID: \"41fbe0e3-2435-4381-940e-508116343854\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5p5wl"
Jul 28 20:48:20.945343 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.945281    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs5ss\" (UniqueName: \"kubernetes.io/projected/1aeaf8a7-0733-428a-81d7-6f57520e2936-kube-api-access-zs5ss\") pod \"cluster-samples-operator-6748489656-lblk5\" (UID: \"1aeaf8a7-0733-428a-81d7-6f57520e2936\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"
Jul 28 20:48:20.950106 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.950086    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"]
Jul 28 20:48:20.963950 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:20.963926    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz6vv\" (UniqueName: \"kubernetes.io/projected/41fbe0e3-2435-4381-940e-508116343854-kube-api-access-gz6vv\") pod \"volume-data-source-validator-549c6d54cf-5p5wl\" (UID: \"41fbe0e3-2435-4381-940e-508116343854\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5p5wl"
Jul 28 20:48:21.050447 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.050407    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lblk5\" (UID: \"1aeaf8a7-0733-428a-81d7-6f57520e2936\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"
Jul 28 20:48:21.050668 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.050510    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs5ss\" (UniqueName: \"kubernetes.io/projected/1aeaf8a7-0733-428a-81d7-6f57520e2936-kube-api-access-zs5ss\") pod \"cluster-samples-operator-6748489656-lblk5\" (UID: \"1aeaf8a7-0733-428a-81d7-6f57520e2936\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"
Jul 28 20:48:21.050668 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:21.050624    2537 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 20:48:21.050789 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:21.050700    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls podName:1aeaf8a7-0733-428a-81d7-6f57520e2936 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:21.550679879 +0000 UTC m=+62.168261766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls") pod "cluster-samples-operator-6748489656-lblk5" (UID: "1aeaf8a7-0733-428a-81d7-6f57520e2936") : secret "samples-operator-tls" not found
Jul 28 20:48:21.051438 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.051414    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-mpflg"]
Jul 28 20:48:21.054148 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.054134    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.059894 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.059765    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 20:48:21.062935 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.062912    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 28 20:48:21.062935 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.062913    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 28 20:48:21.063067 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.063000    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 20:48:21.063067 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.063023    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d4rjl\""
Jul 28 20:48:21.066977 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.066960    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-hv4ql"]
Jul 28 20:48:21.069422 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.069405    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6d8755fdbc-w6k7x"]
Jul 28 20:48:21.069545 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.069530    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hv4ql"
Jul 28 20:48:21.070275 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.070260    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 28 20:48:21.072015 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.072001    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.075031 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.075014    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs5ss\" (UniqueName: \"kubernetes.io/projected/1aeaf8a7-0733-428a-81d7-6f57520e2936-kube-api-access-zs5ss\") pod \"cluster-samples-operator-6748489656-lblk5\" (UID: \"1aeaf8a7-0733-428a-81d7-6f57520e2936\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"
Jul 28 20:48:21.075490 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.075475    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 20:48:21.075721 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.075703    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 20:48:21.075915 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.075901    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mf8m8\""
Jul 28 20:48:21.076012 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.075994    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 20:48:21.076120 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.076094    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 20:48:21.077181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.077160    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 20:48:21.077280 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.077183    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 20:48:21.077280 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.077229    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-t6s6f\""
Jul 28 20:48:21.077828 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.077810    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-mpflg"]
Jul 28 20:48:21.086764 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.086746    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-hv4ql"]
Jul 28 20:48:21.098547 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.098527    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d8755fdbc-w6k7x"]
Jul 28 20:48:21.140322 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.140301    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5p5wl"
Jul 28 20:48:21.151336 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.151315    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5vcj\" (UniqueName: \"kubernetes.io/projected/8956f966-48b7-4f80-8f92-22c03d1287f9-kube-api-access-w5vcj\") pod \"network-check-source-7d88cb55d8-hv4ql\" (UID: \"8956f966-48b7-4f80-8f92-22c03d1287f9\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hv4ql"
Jul 28 20:48:21.151419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.151342    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.151419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.151362    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-stats-auth\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.151419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.151376    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97bvb\" (UniqueName: \"kubernetes.io/projected/8260b33a-e496-4638-996c-7f25da54964f-kube-api-access-97bvb\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.151419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.151400    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cd1b546-a068-41fc-9b32-ff5c2295a654-tmp\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.151604 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.151476    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cd1b546-a068-41fc-9b32-ff5c2295a654-service-ca-bundle\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.151604 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.151527    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5cd1b546-a068-41fc-9b32-ff5c2295a654-snapshots\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.151604 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.151564    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmc6l\" (UniqueName: \"kubernetes.io/projected/5cd1b546-a068-41fc-9b32-ff5c2295a654-kube-api-access-vmc6l\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.151604 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.151596    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cd1b546-a068-41fc-9b32-ff5c2295a654-serving-cert\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.151766 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.151624    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.151766 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.151664    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cd1b546-a068-41fc-9b32-ff5c2295a654-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.151766 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.151704    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-default-certificate\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.229169 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.229123    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:21.252007 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.251980    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cd1b546-a068-41fc-9b32-ff5c2295a654-tmp\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.252132 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.252013    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cd1b546-a068-41fc-9b32-ff5c2295a654-service-ca-bundle\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.252132 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.252038    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5cd1b546-a068-41fc-9b32-ff5c2295a654-snapshots\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.252132 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.252063    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmc6l\" (UniqueName: \"kubernetes.io/projected/5cd1b546-a068-41fc-9b32-ff5c2295a654-kube-api-access-vmc6l\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.252132 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.252093    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cd1b546-a068-41fc-9b32-ff5c2295a654-serving-cert\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.252132 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.252122    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.252350 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.252165    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cd1b546-a068-41fc-9b32-ff5c2295a654-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.252350 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.252219    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-default-certificate\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.252350 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:21.252276    2537 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 20:48:21.252488 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:21.252372    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs podName:8260b33a-e496-4638-996c-7f25da54964f nodeName:}" failed. No retries permitted until 2026-07-28 20:48:21.752350105 +0000 UTC m=+62.369931996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs") pod "router-default-6d8755fdbc-w6k7x" (UID: "8260b33a-e496-4638-996c-7f25da54964f") : secret "router-metrics-certs-default" not found
Jul 28 20:48:21.252488 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.252431    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cd1b546-a068-41fc-9b32-ff5c2295a654-tmp\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.252488 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.252281    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5vcj\" (UniqueName: \"kubernetes.io/projected/8956f966-48b7-4f80-8f92-22c03d1287f9-kube-api-access-w5vcj\") pod \"network-check-source-7d88cb55d8-hv4ql\" (UID: \"8956f966-48b7-4f80-8f92-22c03d1287f9\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hv4ql"
Jul 28 20:48:21.252634 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.252563    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.252634 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.252602    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-stats-auth\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.252634 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.252629    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97bvb\" (UniqueName: \"kubernetes.io/projected/8260b33a-e496-4638-996c-7f25da54964f-kube-api-access-97bvb\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.252759 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:21.252744    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle podName:8260b33a-e496-4638-996c-7f25da54964f nodeName:}" failed. No retries permitted until 2026-07-28 20:48:21.752724251 +0000 UTC m=+62.370306154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle") pod "router-default-6d8755fdbc-w6k7x" (UID: "8260b33a-e496-4638-996c-7f25da54964f") : configmap references non-existent config key: service-ca.crt
Jul 28 20:48:21.252824 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.252771    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5cd1b546-a068-41fc-9b32-ff5c2295a654-snapshots\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.253118 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.253096    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cd1b546-a068-41fc-9b32-ff5c2295a654-service-ca-bundle\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.253539 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.253511    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cd1b546-a068-41fc-9b32-ff5c2295a654-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.254968 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.254885    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-default-certificate\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.254968 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.254929    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-stats-auth\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.255140 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.255124    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cd1b546-a068-41fc-9b32-ff5c2295a654-serving-cert\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.264356 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.264336    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 20:48:21.264798 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.264778    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97bvb\" (UniqueName: \"kubernetes.io/projected/8260b33a-e496-4638-996c-7f25da54964f-kube-api-access-97bvb\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.264888 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.264847    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmc6l\" (UniqueName: \"kubernetes.io/projected/5cd1b546-a068-41fc-9b32-ff5c2295a654-kube-api-access-vmc6l\") pod \"insights-operator-5885fdb759-mpflg\" (UID: \"5cd1b546-a068-41fc-9b32-ff5c2295a654\") " pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.275551 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.275535    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 20:48:21.285040 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.285007    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5vcj\" (UniqueName: \"kubernetes.io/projected/8956f966-48b7-4f80-8f92-22c03d1287f9-kube-api-access-w5vcj\") pod \"network-check-source-7d88cb55d8-hv4ql\" (UID: \"8956f966-48b7-4f80-8f92-22c03d1287f9\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hv4ql"
Jul 28 20:48:21.287341 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.287320    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5p5wl"]
Jul 28 20:48:21.290092 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:48:21.290074    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fbe0e3_2435_4381_940e_508116343854.slice/crio-0f7adad5f887a400507d169a21898f735b59f9eee2677835f73f9d99c25962c9 WatchSource:0}: Error finding container 0f7adad5f887a400507d169a21898f735b59f9eee2677835f73f9d99c25962c9: Status 404 returned error can't find the container with id 0f7adad5f887a400507d169a21898f735b59f9eee2677835f73f9d99c25962c9
Jul 28 20:48:21.295202 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.295183    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:21.362448 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.362395    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-mpflg"
Jul 28 20:48:21.384284 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.384259    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hv4ql"
Jul 28 20:48:21.486581 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.486549    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-mpflg"]
Jul 28 20:48:21.491535 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:48:21.491509    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd1b546_a068_41fc_9b32_ff5c2295a654.slice/crio-45a3c2e6a3978a7b938628ad69ce04642b925cf3ee9782368d9dd7045fb594b6 WatchSource:0}: Error finding container 45a3c2e6a3978a7b938628ad69ce04642b925cf3ee9782368d9dd7045fb594b6: Status 404 returned error can't find the container with id 45a3c2e6a3978a7b938628ad69ce04642b925cf3ee9782368d9dd7045fb594b6
Jul 28 20:48:21.507936 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.507913    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-hv4ql"]
Jul 28 20:48:21.510379 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:48:21.510357    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8956f966_48b7_4f80_8f92_22c03d1287f9.slice/crio-3ccd20a8568b065001ef746b0f1ad14f95f39c370b15c3ae375505be66074a84 WatchSource:0}: Error finding container 3ccd20a8568b065001ef746b0f1ad14f95f39c370b15c3ae375505be66074a84: Status 404 returned error can't find the container with id 3ccd20a8568b065001ef746b0f1ad14f95f39c370b15c3ae375505be66074a84
Jul 28 20:48:21.555365 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.555344    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lblk5\" (UID: \"1aeaf8a7-0733-428a-81d7-6f57520e2936\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"
Jul 28 20:48:21.555452 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:21.555433    2537 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 20:48:21.555494 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:21.555476    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls podName:1aeaf8a7-0733-428a-81d7-6f57520e2936 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:22.555463376 +0000 UTC m=+63.173045257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls") pod "cluster-samples-operator-6748489656-lblk5" (UID: "1aeaf8a7-0733-428a-81d7-6f57520e2936") : secret "samples-operator-tls" not found
Jul 28 20:48:21.757168 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.757095    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.757168 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:21.757146    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:21.757326 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:21.757228    2537 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 20:48:21.757326 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:21.757270    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle podName:8260b33a-e496-4638-996c-7f25da54964f nodeName:}" failed. No retries permitted until 2026-07-28 20:48:22.757252328 +0000 UTC m=+63.374834208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle") pod "router-default-6d8755fdbc-w6k7x" (UID: "8260b33a-e496-4638-996c-7f25da54964f") : configmap references non-existent config key: service-ca.crt
Jul 28 20:48:21.757326 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:21.757290    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs podName:8260b33a-e496-4638-996c-7f25da54964f nodeName:}" failed. No retries permitted until 2026-07-28 20:48:22.757283779 +0000 UTC m=+63.374865658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs") pod "router-default-6d8755fdbc-w6k7x" (UID: "8260b33a-e496-4638-996c-7f25da54964f") : secret "router-metrics-certs-default" not found
Jul 28 20:48:22.258183 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:22.258133    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5p5wl" event={"ID":"41fbe0e3-2435-4381-940e-508116343854","Type":"ContainerStarted","Data":"0f7adad5f887a400507d169a21898f735b59f9eee2677835f73f9d99c25962c9"}
Jul 28 20:48:22.260509 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:22.260462    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mpflg" event={"ID":"5cd1b546-a068-41fc-9b32-ff5c2295a654","Type":"ContainerStarted","Data":"45a3c2e6a3978a7b938628ad69ce04642b925cf3ee9782368d9dd7045fb594b6"}
Jul 28 20:48:22.262532 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:22.262502    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hv4ql" event={"ID":"8956f966-48b7-4f80-8f92-22c03d1287f9","Type":"ContainerStarted","Data":"3ccd20a8568b065001ef746b0f1ad14f95f39c370b15c3ae375505be66074a84"}
Jul 28 20:48:22.564429 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:22.564389    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lblk5\" (UID: \"1aeaf8a7-0733-428a-81d7-6f57520e2936\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"
Jul 28 20:48:22.564606 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:22.564555    2537 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 20:48:22.564666 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:22.564625    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls podName:1aeaf8a7-0733-428a-81d7-6f57520e2936 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:24.564609687 +0000 UTC m=+65.182191572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls") pod "cluster-samples-operator-6748489656-lblk5" (UID: "1aeaf8a7-0733-428a-81d7-6f57520e2936") : secret "samples-operator-tls" not found
Jul 28 20:48:22.765547 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:22.765516    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:22.765685 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:22.765599    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:22.765727 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:22.765706    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle podName:8260b33a-e496-4638-996c-7f25da54964f nodeName:}" failed. No retries permitted until 2026-07-28 20:48:24.765691461 +0000 UTC m=+65.383273341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle") pod "router-default-6d8755fdbc-w6k7x" (UID: "8260b33a-e496-4638-996c-7f25da54964f") : configmap references non-existent config key: service-ca.crt
Jul 28 20:48:22.766093 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:22.766070    2537 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 20:48:22.766198 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:22.766127    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs podName:8260b33a-e496-4638-996c-7f25da54964f nodeName:}" failed. No retries permitted until 2026-07-28 20:48:24.766112539 +0000 UTC m=+65.383694424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs") pod "router-default-6d8755fdbc-w6k7x" (UID: "8260b33a-e496-4638-996c-7f25da54964f") : secret "router-metrics-certs-default" not found
Jul 28 20:48:23.266938 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:23.266901    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5p5wl" event={"ID":"41fbe0e3-2435-4381-940e-508116343854","Type":"ContainerStarted","Data":"ec00f463aa42dea3a56c42ae96bff844bf35f27e527b86360068a3dc593a381e"}
Jul 28 20:48:23.286314 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:23.286103    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-5p5wl" podStartSLOduration=1.960391507 podStartE2EDuration="3.286082841s" totalImagesPullingTime="1.325691334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:23.284822662 +0000 UTC m=+63.902404564" watchObservedRunningTime="2026-07-28 20:48:23.286082841 +0000 UTC m=+63.903664746"
Jul 28 20:48:23.378560 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:23.378504    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:23.444816 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:23.444779    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:24.580324 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:24.580288    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lblk5\" (UID: \"1aeaf8a7-0733-428a-81d7-6f57520e2936\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"
Jul 28 20:48:24.580695 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:24.580446    2537 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 20:48:24.580695 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:24.580508    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls podName:1aeaf8a7-0733-428a-81d7-6f57520e2936 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:28.580489412 +0000 UTC m=+69.198071307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls") pod "cluster-samples-operator-6748489656-lblk5" (UID: "1aeaf8a7-0733-428a-81d7-6f57520e2936") : secret "samples-operator-tls" not found
Jul 28 20:48:24.782183 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:24.782145    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:24.782384 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:24.782204    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:24.782384 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:24.782332    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle podName:8260b33a-e496-4638-996c-7f25da54964f nodeName:}" failed. No retries permitted until 2026-07-28 20:48:28.782309675 +0000 UTC m=+69.399891560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle") pod "router-default-6d8755fdbc-w6k7x" (UID: "8260b33a-e496-4638-996c-7f25da54964f") : configmap references non-existent config key: service-ca.crt
Jul 28 20:48:24.782384 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:24.782344    2537 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 20:48:24.782549 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:24.782390    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs podName:8260b33a-e496-4638-996c-7f25da54964f nodeName:}" failed. No retries permitted until 2026-07-28 20:48:28.782374276 +0000 UTC m=+69.399956156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs") pod "router-default-6d8755fdbc-w6k7x" (UID: "8260b33a-e496-4638-996c-7f25da54964f") : secret "router-metrics-certs-default" not found
Jul 28 20:48:25.274209 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.274173    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mpflg" event={"ID":"5cd1b546-a068-41fc-9b32-ff5c2295a654","Type":"ContainerStarted","Data":"98936edb01d54d110a5913015b0b0d1aad5ba13d5bb594ccad8e110d91353dfa"}
Jul 28 20:48:25.275656 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.275631    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hv4ql" event={"ID":"8956f966-48b7-4f80-8f92-22c03d1287f9","Type":"ContainerStarted","Data":"e111ead1fea811ef29f019b0ddce38009568d8668208ac0d72851e5fe32aaffd"}
Jul 28 20:48:25.298201 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.298140    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-mpflg" podStartSLOduration=1.168246459 podStartE2EDuration="4.298126874s" totalImagesPullingTime="3.129880415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:25.296949722 +0000 UTC m=+65.914531621" watchObservedRunningTime="2026-07-28 20:48:25.298126874 +0000 UTC m=+65.915708795"
Jul 28 20:48:25.315007 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.314962    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hv4ql" podStartSLOduration=1.196630028 podStartE2EDuration="4.314945442s" totalImagesPullingTime="3.118315414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:25.314164802 +0000 UTC m=+65.931746705" watchObservedRunningTime="2026-07-28 20:48:25.314945442 +0000 UTC m=+65.932527344"
Jul 28 20:48:25.586434 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.586401    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs\") pod \"network-metrics-daemon-9qhw8\" (UID: \"5e0b9b3f-4257-4183-90eb-1b7102f3606d\") " pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:48:25.594232 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.594204    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 20:48:25.596997 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:25.596980    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 20:48:25.597103 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:25.597050    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs podName:5e0b9b3f-4257-4183-90eb-1b7102f3606d nodeName:}" failed. No retries permitted until 2026-07-28 20:49:29.597023263 +0000 UTC m=+130.214605159 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs") pod "network-metrics-daemon-9qhw8" (UID: "5e0b9b3f-4257-4183-90eb-1b7102f3606d") : secret "metrics-daemon-secret" not found
Jul 28 20:48:25.687658 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.687630    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:48:25.687788 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.687675    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mzq\" (UniqueName: \"kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq\") pod \"network-check-target-lw7l6\" (UID: \"59627965-db98-4b3f-82ce-b9621a1eef88\") " pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:48:25.690157 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.690135    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4mzq\" (UniqueName: \"kubernetes.io/projected/59627965-db98-4b3f-82ce-b9621a1eef88-kube-api-access-m4mzq\") pod \"network-check-target-lw7l6\" (UID: \"59627965-db98-4b3f-82ce-b9621a1eef88\") " pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:48:25.691066 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.691045    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 20:48:25.700128 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.700105    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415a6e79-19ba-4beb-ab92-8b2b739d63a4-original-pull-secret\") pod \"global-pull-secret-syncer-xx9kc\" (UID: \"415a6e79-19ba-4beb-ab92-8b2b739d63a4\") " pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:48:25.708740 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.708675    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:25.731587 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.731562    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j52pw\""
Jul 28 20:48:25.733689 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.733668    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xx9kc"
Jul 28 20:48:25.738745 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.738723    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:48:25.813941 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.812667    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gprrj"]
Jul 28 20:48:25.818492 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.817719    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:25.820074 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.819996    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:25.822628 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.822085    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 20:48:25.822628 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.822438    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9jg6j\""
Jul 28 20:48:25.822628 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.822566    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 20:48:25.836004 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.835929    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gprrj"]
Jul 28 20:48:25.891042 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.891008    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55e48be1-28c7-409d-baf5-939aeeff7d34-crio-socket\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:25.891200 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.891071    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:25.891200 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.891115    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55e48be1-28c7-409d-baf5-939aeeff7d34-data-volume\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:25.891200 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.891135    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:25.891328 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.891217    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xmx\" (UniqueName: \"kubernetes.io/projected/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-api-access-f4xmx\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:25.896657 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.896626    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xx9kc"]
Jul 28 20:48:25.900984 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:48:25.900954    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415a6e79_19ba_4beb_ab92_8b2b739d63a4.slice/crio-008cd5c0b7e9f89df8ee58eac02d2e7f7652ad8f91716b4ff965e7d20aa308b3 WatchSource:0}: Error finding container 008cd5c0b7e9f89df8ee58eac02d2e7f7652ad8f91716b4ff965e7d20aa308b3: Status 404 returned error can't find the container with id 008cd5c0b7e9f89df8ee58eac02d2e7f7652ad8f91716b4ff965e7d20aa308b3
Jul 28 20:48:25.901541 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.901519    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:25.914736 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.914715    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lw7l6"]
Jul 28 20:48:25.917663 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:48:25.917638    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59627965_db98_4b3f_82ce_b9621a1eef88.slice/crio-59d88f44bd24c08e3c1ecc70c4c3f615cd3b028d4839c768e5bea1a3f40612c9 WatchSource:0}: Error finding container 59d88f44bd24c08e3c1ecc70c4c3f615cd3b028d4839c768e5bea1a3f40612c9: Status 404 returned error can't find the container with id 59d88f44bd24c08e3c1ecc70c4c3f615cd3b028d4839c768e5bea1a3f40612c9
Jul 28 20:48:25.992731 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.992709    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:25.992806 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.992754    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55e48be1-28c7-409d-baf5-939aeeff7d34-data-volume\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:25.992806 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.992773    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:25.992806 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.992798    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xmx\" (UniqueName: \"kubernetes.io/projected/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-api-access-f4xmx\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:25.992948 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:25.992826    2537 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:48:25.992948 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.992841    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55e48be1-28c7-409d-baf5-939aeeff7d34-crio-socket\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:25.992948 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:25.992929    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm podName:55e48be1-28c7-409d-baf5-939aeeff7d34 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:26.492909464 +0000 UTC m=+67.110491360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gprrj" (UID: "55e48be1-28c7-409d-baf5-939aeeff7d34") : configmap "kube-rbac-proxy" not found
Jul 28 20:48:25.992948 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:25.992935    2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:25.993134 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.992948    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55e48be1-28c7-409d-baf5-939aeeff7d34-crio-socket\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:25.993134 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:25.992985    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls podName:55e48be1-28c7-409d-baf5-939aeeff7d34 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:26.492970219 +0000 UTC m=+67.110552110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gprrj" (UID: "55e48be1-28c7-409d-baf5-939aeeff7d34") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:25.993134 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:25.993052    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55e48be1-28c7-409d-baf5-939aeeff7d34-data-volume\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:26.005949 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:26.005927    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xmx\" (UniqueName: \"kubernetes.io/projected/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-api-access-f4xmx\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:26.280320 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:26.280241    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lw7l6" event={"ID":"59627965-db98-4b3f-82ce-b9621a1eef88","Type":"ContainerStarted","Data":"a62ee2f203aa4d07572ee656a7b12babb8ef43ec8e6bb3c2b1f605d55b33d90b"}
Jul 28 20:48:26.280320 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:26.280296    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lw7l6" event={"ID":"59627965-db98-4b3f-82ce-b9621a1eef88","Type":"ContainerStarted","Data":"59d88f44bd24c08e3c1ecc70c4c3f615cd3b028d4839c768e5bea1a3f40612c9"}
Jul 28 20:48:26.280564 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:26.280332    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:48:26.281345 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:26.281319    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xx9kc" event={"ID":"415a6e79-19ba-4beb-ab92-8b2b739d63a4","Type":"ContainerStarted","Data":"008cd5c0b7e9f89df8ee58eac02d2e7f7652ad8f91716b4ff965e7d20aa308b3"}
Jul 28 20:48:26.299361 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:26.299313    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lw7l6" podStartSLOduration=66.299296725 podStartE2EDuration="1m6.299296725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:26.298438962 +0000 UTC m=+66.916020863" watchObservedRunningTime="2026-07-28 20:48:26.299296725 +0000 UTC m=+66.916878625"
Jul 28 20:48:26.497121 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:26.497087    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:26.497332 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:26.497141    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:26.497332 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:26.497230    2537 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:48:26.497332 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:26.497303    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm podName:55e48be1-28c7-409d-baf5-939aeeff7d34 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:27.497283839 +0000 UTC m=+68.114865734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gprrj" (UID: "55e48be1-28c7-409d-baf5-939aeeff7d34") : configmap "kube-rbac-proxy" not found
Jul 28 20:48:26.497507 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:26.497362    2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:26.497507 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:26.497422    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls podName:55e48be1-28c7-409d-baf5-939aeeff7d34 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:27.497405857 +0000 UTC m=+68.114987739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gprrj" (UID: "55e48be1-28c7-409d-baf5-939aeeff7d34") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:26.874173 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:26.874139    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:26.964888 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:26.964846    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:27.040115 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:27.040086    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:27.101616 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:27.101586    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:48:27.101763 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:27.101740    2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:48:27.101817 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:27.101803    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls podName:0b960510-a109-42a8-99d8-dad53a778237 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:59.101786831 +0000 UTC m=+99.719368712 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls") pod "dns-default-mrnqh" (UID: "0b960510-a109-42a8-99d8-dad53a778237") : secret "dns-default-metrics-tls" not found
Jul 28 20:48:27.505528 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:27.505492    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:27.505718 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:27.505547    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:27.505718 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:27.505626    2537 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:48:27.505718 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:27.505692    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm podName:55e48be1-28c7-409d-baf5-939aeeff7d34 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:29.505675186 +0000 UTC m=+70.123257089 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gprrj" (UID: "55e48be1-28c7-409d-baf5-939aeeff7d34") : configmap "kube-rbac-proxy" not found
Jul 28 20:48:27.505718 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:27.505709    2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:27.505947 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:27.505743    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls podName:55e48be1-28c7-409d-baf5-939aeeff7d34 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:29.505733425 +0000 UTC m=+70.123315305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gprrj" (UID: "55e48be1-28c7-409d-baf5-939aeeff7d34") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:27.723151 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:27.723118    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:27.952174 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:27.952143    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tk54b_4421ad59-d310-4fce-b6f2-5518ede532fa/dns-node-resolver/0.log"
Jul 28 20:48:28.187753 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:28.187721    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:28.289085 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:28.288985    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:28.468586 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:28.468539    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:28.552718 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:28.552602    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:28.616331 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:28.616304    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lblk5\" (UID: \"1aeaf8a7-0733-428a-81d7-6f57520e2936\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"
Jul 28 20:48:28.616490 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:28.616460    2537 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 20:48:28.616546 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:28.616521    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls podName:1aeaf8a7-0733-428a-81d7-6f57520e2936 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:36.616503726 +0000 UTC m=+77.234085606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls") pod "cluster-samples-operator-6748489656-lblk5" (UID: "1aeaf8a7-0733-428a-81d7-6f57520e2936") : secret "samples-operator-tls" not found
Jul 28 20:48:28.714044 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:28.714009    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:28.751652 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:28.751607    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6vhdp_19ba6b7f-59fc-4288-aef0-a479c989e6d7/node-ca/0.log"
Jul 28 20:48:28.818244 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:28.818210    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:28.818416 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:28.818318    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:28.818416 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:28.818364    2537 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 20:48:28.818516 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:28.818430    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs podName:8260b33a-e496-4638-996c-7f25da54964f nodeName:}" failed. No retries permitted until 2026-07-28 20:48:36.818410124 +0000 UTC m=+77.435992008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs") pod "router-default-6d8755fdbc-w6k7x" (UID: "8260b33a-e496-4638-996c-7f25da54964f") : secret "router-metrics-certs-default" not found
Jul 28 20:48:28.818516 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:28.818450    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle podName:8260b33a-e496-4638-996c-7f25da54964f nodeName:}" failed. No retries permitted until 2026-07-28 20:48:36.818440262 +0000 UTC m=+77.436022144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle") pod "router-default-6d8755fdbc-w6k7x" (UID: "8260b33a-e496-4638-996c-7f25da54964f") : configmap references non-existent config key: service-ca.crt
Jul 28 20:48:29.523837 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:29.523791    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:29.524331 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:29.523952    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:29.524331 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:29.523981    2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:29.524331 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:29.524038    2537 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:48:29.524331 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:29.524066    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls podName:55e48be1-28c7-409d-baf5-939aeeff7d34 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:33.524044053 +0000 UTC m=+74.141625958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gprrj" (UID: "55e48be1-28c7-409d-baf5-939aeeff7d34") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:29.524331 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:29.524090    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm podName:55e48be1-28c7-409d-baf5-939aeeff7d34 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:33.524076389 +0000 UTC m=+74.141658269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gprrj" (UID: "55e48be1-28c7-409d-baf5-939aeeff7d34") : configmap "kube-rbac-proxy" not found
Jul 28 20:48:29.753376 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:29.753349    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r2g2c_8083d69e-72cb-48d1-91bb-286ee0b4141d/serve-healthcheck-canary/0.log"
Jul 28 20:48:30.082463 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:30.082426    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:30.144042 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:30.144011    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:30.296159 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:30.296126    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xx9kc" event={"ID":"415a6e79-19ba-4beb-ab92-8b2b739d63a4","Type":"ContainerStarted","Data":"a5849ea6fd10d11e233acb78bfd48a20a339e7ffce344a720f0213857f047fb3"}
Jul 28 20:48:30.315116 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:30.315076    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xx9kc" podStartSLOduration=66.611906497 podStartE2EDuration="1m10.315062277s" totalImagesPullingTime="3.70315578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:30.314788505 +0000 UTC m=+70.932370406" watchObservedRunningTime="2026-07-28 20:48:30.315062277 +0000 UTC m=+70.932644170"
Jul 28 20:48:31.050557 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:31.050529    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:31.151820 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:31.151788    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:31.626451 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:31.626414    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:31.784067 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:31.784040    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:31.899806 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:31.899735    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:31.988524 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:31.988501    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:32.067722 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:32.067692    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:32.512412 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:32.512385    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:32.615623 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:32.615594    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:33.136676 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:33.136647    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:33.529445 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:33.529371    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:33.557702 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:33.557678    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:33.557788 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:33.557772    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:33.557835 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:33.557807    2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:33.557895 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:33.557870    2537 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:48:33.557895 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:33.557880    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls podName:55e48be1-28c7-409d-baf5-939aeeff7d34 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:41.557848507 +0000 UTC m=+82.175430388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gprrj" (UID: "55e48be1-28c7-409d-baf5-939aeeff7d34") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:33.557967 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:33.557923    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm podName:55e48be1-28c7-409d-baf5-939aeeff7d34 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:41.557908408 +0000 UTC m=+82.175490288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gprrj" (UID: "55e48be1-28c7-409d-baf5-939aeeff7d34") : configmap "kube-rbac-proxy" not found
Jul 28 20:48:33.924258 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:33.924226    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:34.131241 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:34.131204    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:34.530468 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:34.530418    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:34.935527 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:34.935498    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:35.284488 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:35.284400    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:35.381972 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:35.381942    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:35.454094 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:35.454055    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:35.737378 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:35.737346    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:35.931499 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:35.931467    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:36.035327 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:36.035258    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:36.134125 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:36.134097    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:36.329653 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:36.329626    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:36.679555 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:36.679481    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lblk5\" (UID: \"1aeaf8a7-0733-428a-81d7-6f57520e2936\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"
Jul 28 20:48:36.681799 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:36.681780    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aeaf8a7-0733-428a-81d7-6f57520e2936-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lblk5\" (UID: \"1aeaf8a7-0733-428a-81d7-6f57520e2936\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"
Jul 28 20:48:36.844082 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:36.844051    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"
Jul 28 20:48:36.881350 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:36.881313    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:36.881493 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:36.881383    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:36.881561 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:36.881496    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle podName:8260b33a-e496-4638-996c-7f25da54964f nodeName:}" failed. No retries permitted until 2026-07-28 20:48:52.881479764 +0000 UTC m=+93.499061647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle") pod "router-default-6d8755fdbc-w6k7x" (UID: "8260b33a-e496-4638-996c-7f25da54964f") : configmap references non-existent config key: service-ca.crt
Jul 28 20:48:36.883923 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:36.883897    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8260b33a-e496-4638-996c-7f25da54964f-metrics-certs\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:36.974523 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:36.974453    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5"]
Jul 28 20:48:37.318293 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:37.318259    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5" event={"ID":"1aeaf8a7-0733-428a-81d7-6f57520e2936","Type":"ContainerStarted","Data":"8a239a771c57e6e6983f6a576ab14db5deafb3ac504c1f11617a690abcd612ff"}
Jul 28 20:48:37.322798 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:37.322774    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:37.564157 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:37.564120    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:37.725018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:37.724937    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:37.921816 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:37.921777    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:38.121283 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:38.121251    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:38.530205 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:38.530177    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:39.006310 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:39.006282    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:39.131826 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:39.131795    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:39.326049 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:39.326014    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:39.327506 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:39.327484    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5" event={"ID":"1aeaf8a7-0733-428a-81d7-6f57520e2936","Type":"ContainerStarted","Data":"95e7eb9fdaf0f36afab30d5c0a9897b6f5949ca4dd5133e582dbf3b2bec4dd85"}
Jul 28 20:48:39.327605 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:39.327533    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5" event={"ID":"1aeaf8a7-0733-428a-81d7-6f57520e2936","Type":"ContainerStarted","Data":"44901662cc4d946b94951bccb11705bb6150f4c0af1f1619875e90c302aebbcb"}
Jul 28 20:48:39.346844 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:39.346804    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lblk5" podStartSLOduration=17.852523076 podStartE2EDuration="19.346791946s" totalImagesPullingTime="1.49426887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:39.345965261 +0000 UTC m=+79.963547175" watchObservedRunningTime="2026-07-28 20:48:39.346791946 +0000 UTC m=+79.964373845"
Jul 28 20:48:39.733822 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:39.733746    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:40.528185 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:40.528144    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:40.986625 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:40.986580    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:41.128848 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:41.128813    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:41.325400 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:41.325375    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:41.620019 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:41.619947    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:41.620019 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:41.620009    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:41.620390 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:41.620094    2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:41.620390 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:48:41.620163    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls podName:55e48be1-28c7-409d-baf5-939aeeff7d34 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:57.62014828 +0000 UTC m=+98.237730160 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gprrj" (UID: "55e48be1-28c7-409d-baf5-939aeeff7d34") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:41.620390 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:41.620269    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e48be1-28c7-409d-baf5-939aeeff7d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:41.768948 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:41.768919    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:42.128835 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:42.128810    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:42.325918 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:42.325878    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:43.494935 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:43.494900    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:43.595759 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:43.595732    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:43.654684 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:43.654660    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:46.218468 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:46.218421    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:46.314253 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:46.314225    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:46.376027 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:46.376000    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:46.489706 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:46.489647    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:46.548428 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:46.548401    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:46.726240 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:46.726210    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:46.896797 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:46.896771    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:46.955905 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:46.955883    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:47.153072 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:47.153008    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:47.300663 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:47.300634    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:47.712044 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:47.712013    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:47.908161 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:47.908132    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:48.176454 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:48.176428    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:49.152448 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:49.152420    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:49.207660 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:49.207630    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:52.900716 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:52.900685    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:52.901248 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:52.901229    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8260b33a-e496-4638-996c-7f25da54964f-service-ca-bundle\") pod \"router-default-6d8755fdbc-w6k7x\" (UID: \"8260b33a-e496-4638-996c-7f25da54964f\") " pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:53.188807 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:53.188725    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:53.321770 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:53.321740    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d8755fdbc-w6k7x"]
Jul 28 20:48:53.324954 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:48:53.324925    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8260b33a_e496_4638_996c_7f25da54964f.slice/crio-bf442d390055f0e00f3da3f2827e3f5bd1c667ec871bd959e3c6e73a0f420523 WatchSource:0}: Error finding container bf442d390055f0e00f3da3f2827e3f5bd1c667ec871bd959e3c6e73a0f420523: Status 404 returned error can't find the container with id bf442d390055f0e00f3da3f2827e3f5bd1c667ec871bd959e3c6e73a0f420523
Jul 28 20:48:53.365977 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:53.365954    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d8755fdbc-w6k7x" event={"ID":"8260b33a-e496-4638-996c-7f25da54964f","Type":"ContainerStarted","Data":"bf442d390055f0e00f3da3f2827e3f5bd1c667ec871bd959e3c6e73a0f420523"}
Jul 28 20:48:54.370675 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.370638    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d8755fdbc-w6k7x" event={"ID":"8260b33a-e496-4638-996c-7f25da54964f","Type":"ContainerStarted","Data":"084f38fa4e8f1e9adbcb61b3bd058c9ba4122910054235a043f2dc976e0968af"}
Jul 28 20:48:54.398228 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.398178    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6d8755fdbc-w6k7x" podStartSLOduration=33.398162518 podStartE2EDuration="33.398162518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:54.397251058 +0000 UTC m=+95.014832961" watchObservedRunningTime="2026-07-28 20:48:54.398162518 +0000 UTC m=+95.015744420"
Jul 28 20:48:54.527527 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.527498    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"]
Jul 28 20:48:54.530907 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.530888    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.535380 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.535358    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 20:48:54.535487 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.535461    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 20:48:54.535818 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.535798    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 20:48:54.536368 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.536352    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 20:48:54.536454 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.536436    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 20:48:54.536616 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.536598    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 20:48:54.537160 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.537136    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 20:48:54.546286 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.546257    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"]
Jul 28 20:48:54.614387 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.614360    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb173b40-cd26-481c-bc7f-963036eb27f6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.614501 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.614412    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cb173b40-cd26-481c-bc7f-963036eb27f6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.614501 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.614436    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cb173b40-cd26-481c-bc7f-963036eb27f6-hub\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.614501 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.614490    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cb173b40-cd26-481c-bc7f-963036eb27f6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.614596 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.614543    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cb173b40-cd26-481c-bc7f-963036eb27f6-ca\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.614596 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.614571    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7298\" (UniqueName: \"kubernetes.io/projected/cb173b40-cd26-481c-bc7f-963036eb27f6-kube-api-access-l7298\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.654675 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.654610    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:54.715729 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.715703    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb173b40-cd26-481c-bc7f-963036eb27f6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.715835 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.715758    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cb173b40-cd26-481c-bc7f-963036eb27f6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.715835 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.715785    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cb173b40-cd26-481c-bc7f-963036eb27f6-hub\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.715835 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.715808    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cb173b40-cd26-481c-bc7f-963036eb27f6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.716006 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.715844    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cb173b40-cd26-481c-bc7f-963036eb27f6-ca\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.716006 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.715885    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7298\" (UniqueName: \"kubernetes.io/projected/cb173b40-cd26-481c-bc7f-963036eb27f6-kube-api-access-l7298\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.716493 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.716465    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb173b40-cd26-481c-bc7f-963036eb27f6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.718674 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.718654    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cb173b40-cd26-481c-bc7f-963036eb27f6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.718776 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.718677    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cb173b40-cd26-481c-bc7f-963036eb27f6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.718776 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.718677    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cb173b40-cd26-481c-bc7f-963036eb27f6-ca\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.718910 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.718801    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cb173b40-cd26-481c-bc7f-963036eb27f6-hub\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.725272 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.725252    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7298\" (UniqueName: \"kubernetes.io/projected/cb173b40-cd26-481c-bc7f-963036eb27f6-kube-api-access-l7298\") pod \"cluster-proxy-proxy-agent-9f4dcc44b-m8fzn\" (UID: \"cb173b40-cd26-481c-bc7f-963036eb27f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.854747 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.854717    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"
Jul 28 20:48:54.888467 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.888435    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:54.977510 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:54.977480    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn"]
Jul 28 20:48:54.979954 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:48:54.979923    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb173b40_cd26_481c_bc7f_963036eb27f6.slice/crio-ebf681ebc2fb4b04bb587eebf58e849374640a5d8c6753728185d18197fe0d8b WatchSource:0}: Error finding container ebf681ebc2fb4b04bb587eebf58e849374640a5d8c6753728185d18197fe0d8b: Status 404 returned error can't find the container with id ebf681ebc2fb4b04bb587eebf58e849374640a5d8c6753728185d18197fe0d8b
Jul 28 20:48:55.189503 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:55.189426    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:55.191940 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:55.191918    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:55.307639 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:55.307613    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:55.375370 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:55.375337    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn" event={"ID":"cb173b40-cd26-481c-bc7f-963036eb27f6","Type":"ContainerStarted","Data":"ebf681ebc2fb4b04bb587eebf58e849374640a5d8c6753728185d18197fe0d8b"}
Jul 28 20:48:55.375757 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:55.375539    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:55.376774 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:55.376756    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6d8755fdbc-w6k7x"
Jul 28 20:48:55.515821 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:55.515779    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:55.807004 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:55.806923    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:56.847726 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:56.847692    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:56.948280 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:56.948241    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:57.005832 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:57.005799    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:57.287082 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:57.287005    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lw7l6"
Jul 28 20:48:57.640140 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:57.640099    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:57.642841 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:57.642816    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e48be1-28c7-409d-baf5-939aeeff7d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gprrj\" (UID: \"55e48be1-28c7-409d-baf5-939aeeff7d34\") " pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:57.929986 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:57.929896    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gprrj"
Jul 28 20:48:58.315321 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:58.315291    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gprrj"]
Jul 28 20:48:58.319174 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:48:58.319134    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e48be1_28c7_409d_baf5_939aeeff7d34.slice/crio-63847c1c1a200dcd41f32be694013588c8e04eb3befd82246fbf3da958184787 WatchSource:0}: Error finding container 63847c1c1a200dcd41f32be694013588c8e04eb3befd82246fbf3da958184787: Status 404 returned error can't find the container with id 63847c1c1a200dcd41f32be694013588c8e04eb3befd82246fbf3da958184787
Jul 28 20:48:58.387289 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:58.387261    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gprrj" event={"ID":"55e48be1-28c7-409d-baf5-939aeeff7d34","Type":"ContainerStarted","Data":"63847c1c1a200dcd41f32be694013588c8e04eb3befd82246fbf3da958184787"}
Jul 28 20:48:59.151251 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:59.151207    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:48:59.153745 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:59.153711    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b960510-a109-42a8-99d8-dad53a778237-metrics-tls\") pod \"dns-default-mrnqh\" (UID: \"0b960510-a109-42a8-99d8-dad53a778237\") " pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:48:59.161493 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:59.161467    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ghnpw\""
Jul 28 20:48:59.168915 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:59.168898    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:48:59.596398 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:48:59.596370    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mrnqh"]
Jul 28 20:49:00.396939 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:00.396897    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn" event={"ID":"cb173b40-cd26-481c-bc7f-963036eb27f6","Type":"ContainerStarted","Data":"7f378f833267480f50c38dbd59323cf79106ce4f8bc072060dc996aa9f937401"}
Jul 28 20:49:00.397413 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:00.396957    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn" event={"ID":"cb173b40-cd26-481c-bc7f-963036eb27f6","Type":"ContainerStarted","Data":"7a6d5c1db0de5892796a860f205383148ed7314081898d87409167b1df4e6fae"}
Jul 28 20:49:00.397413 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:00.396979    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn" event={"ID":"cb173b40-cd26-481c-bc7f-963036eb27f6","Type":"ContainerStarted","Data":"399af4c193d5c45640912f8a7fb74e3615fce49826cb9ecf20b2e9bb9d1e2152"}
Jul 28 20:49:00.398584 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:00.398515    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mrnqh" event={"ID":"0b960510-a109-42a8-99d8-dad53a778237","Type":"ContainerStarted","Data":"03cd97824bf65b8b648ed69531509ea6946e0298d4123029609b7086db8e907b"}
Jul 28 20:49:00.421199 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:00.421140    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9f4dcc44b-m8fzn" podStartSLOduration=1.8658351880000001 podStartE2EDuration="6.421121762s" totalImagesPullingTime="4.555286574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:00.420071162 +0000 UTC m=+101.037653065" watchObservedRunningTime="2026-07-28 20:49:00.421121762 +0000 UTC m=+101.038703657"
Jul 28 20:49:00.546606 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:00.546542    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:00.605464 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:00.605430    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:02.331036 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:02.331000    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:02.393663 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:02.393626    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:02.406913 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:02.406886    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mrnqh" event={"ID":"0b960510-a109-42a8-99d8-dad53a778237","Type":"ContainerStarted","Data":"a764f3e746ae2e3e8c71d7a4626f09b529a3978c3fc20f1c8088035cd9059039"}
Jul 28 20:49:02.407047 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:02.406940    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mrnqh" event={"ID":"0b960510-a109-42a8-99d8-dad53a778237","Type":"ContainerStarted","Data":"f0c73d77df2b7f787f2ca25fad0eb2122e7ef842221e74f73393a23f46e81cdd"}
Jul 28 20:49:02.407047 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:02.406989    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:49:02.436008 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:02.435965    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mrnqh" podStartSLOduration=65.661975918 podStartE2EDuration="1m7.43594724s" totalImagesPullingTime="1.773971322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:02.435044299 +0000 UTC m=+103.052626203" watchObservedRunningTime="2026-07-28 20:49:02.43594724 +0000 UTC m=+103.053529143"
Jul 28 20:49:02.568474 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:02.568435    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:02.660537 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:02.660454    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:02.879559 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:02.879524    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:05.136704 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:05.136673    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:05.241915 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:05.241834    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:05.419554 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:05.419520    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gprrj" event={"ID":"55e48be1-28c7-409d-baf5-939aeeff7d34","Type":"ContainerStarted","Data":"2c7d2f2c1b8b1ecd13aa5182d81605917681db8933576584f2fe1599d976fcdb"}
Jul 28 20:49:06.335960 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:06.335918    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:06.394434 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:06.394401    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:08.430812 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:08.430782    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gprrj" event={"ID":"55e48be1-28c7-409d-baf5-939aeeff7d34","Type":"ContainerStarted","Data":"3dbe9ac8ca3b70f8774dc8da70c7d83d7a1455793d624ef405fd7d93602efafc"}
Jul 28 20:49:08.903932 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:08.903881    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:10.733823 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:10.733796    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:10.944066 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:10.944029    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:11.036632 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.036554    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4nmrq"]
Jul 28 20:49:11.043832 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.043793    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:11.044373 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.044343    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"]
Jul 28 20:49:11.044501 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.044477    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.049970 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.049947    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.055455 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.055431    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 20:49:11.056002 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.055976    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 20:49:11.056112 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.056062    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 20:49:11.056175 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.056133    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 20:49:11.056236 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.056181    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 20:49:11.056407 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.056386    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-d2fw6\""
Jul 28 20:49:11.058222 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.058200    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/303a511a-4ac0-4b7d-bc70-e21fde77d433-sys\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.058339 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.058247    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-textfile\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.058339 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.058319    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/303a511a-4ac0-4b7d-bc70-e21fde77d433-root\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.058454 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.058379    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.058454 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.058422    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnkrg\" (UniqueName: \"kubernetes.io/projected/303a511a-4ac0-4b7d-bc70-e21fde77d433-kube-api-access-nnkrg\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.058555 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.058480    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-tls\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.058555 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.058515    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96b558af-774c-4a05-a70f-0ebe768ebeee-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.058555 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.058543    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/96b558af-774c-4a05-a70f-0ebe768ebeee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.058759 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.058573    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/303a511a-4ac0-4b7d-bc70-e21fde77d433-metrics-client-ca\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.058759 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.058652    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-accelerators-collector-config\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.058759 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.058698    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6mr4\" (UniqueName: \"kubernetes.io/projected/96b558af-774c-4a05-a70f-0ebe768ebeee-kube-api-access-b6mr4\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.058759 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.058729    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-wtmp\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.058759 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.058756    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96b558af-774c-4a05-a70f-0ebe768ebeee-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.058981 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.058792    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/96b558af-774c-4a05-a70f-0ebe768ebeee-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.058981 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.058825    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96b558af-774c-4a05-a70f-0ebe768ebeee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.060063 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.059892    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 20:49:11.060509 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.060480    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 20:49:11.060595 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.060574    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 20:49:11.060648 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.060611    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-96xfh\""
Jul 28 20:49:11.060872 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.060839    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 20:49:11.091664 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.091640    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"]
Jul 28 20:49:11.136240 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.136191    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:11.159435 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159402    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-accelerators-collector-config\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.159584 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159449    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6mr4\" (UniqueName: \"kubernetes.io/projected/96b558af-774c-4a05-a70f-0ebe768ebeee-kube-api-access-b6mr4\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.159584 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159502    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-wtmp\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.159584 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159528    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96b558af-774c-4a05-a70f-0ebe768ebeee-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.159584 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159575    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/96b558af-774c-4a05-a70f-0ebe768ebeee-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.159784 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159609    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96b558af-774c-4a05-a70f-0ebe768ebeee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.159784 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159639    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/303a511a-4ac0-4b7d-bc70-e21fde77d433-sys\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.159784 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:49:11.159673    2537 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 28 20:49:11.159784 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159683    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-wtmp\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.159784 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159676    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-textfile\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.159784 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159742    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/303a511a-4ac0-4b7d-bc70-e21fde77d433-sys\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.159784 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:49:11.159750    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b558af-774c-4a05-a70f-0ebe768ebeee-kube-state-metrics-tls podName:96b558af-774c-4a05-a70f-0ebe768ebeee nodeName:}" failed. No retries permitted until 2026-07-28 20:49:11.659729999 +0000 UTC m=+112.277311894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/96b558af-774c-4a05-a70f-0ebe768ebeee-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-qspqh" (UID: "96b558af-774c-4a05-a70f-0ebe768ebeee") : secret "kube-state-metrics-tls" not found
Jul 28 20:49:11.159784 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159786    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/303a511a-4ac0-4b7d-bc70-e21fde77d433-root\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.160191 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159812    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.160191 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159837    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnkrg\" (UniqueName: \"kubernetes.io/projected/303a511a-4ac0-4b7d-bc70-e21fde77d433-kube-api-access-nnkrg\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.160191 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159900    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-tls\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.160191 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159925    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96b558af-774c-4a05-a70f-0ebe768ebeee-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.160191 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159928    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/303a511a-4ac0-4b7d-bc70-e21fde77d433-root\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.160191 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159952    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/96b558af-774c-4a05-a70f-0ebe768ebeee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.160191 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.159985    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/303a511a-4ac0-4b7d-bc70-e21fde77d433-metrics-client-ca\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.160191 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.160045    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/96b558af-774c-4a05-a70f-0ebe768ebeee-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.160191 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.160052    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-accelerators-collector-config\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.160191 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:49:11.160140    2537 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 20:49:11.160191 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:49:11.160201    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-tls podName:303a511a-4ac0-4b7d-bc70-e21fde77d433 nodeName:}" failed. No retries permitted until 2026-07-28 20:49:11.660182792 +0000 UTC m=+112.277764680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-tls") pod "node-exporter-4nmrq" (UID: "303a511a-4ac0-4b7d-bc70-e21fde77d433") : secret "node-exporter-tls" not found
Jul 28 20:49:11.160770 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.160391    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-textfile\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.160770 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.160484    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/303a511a-4ac0-4b7d-bc70-e21fde77d433-metrics-client-ca\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.160770 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.160711    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96b558af-774c-4a05-a70f-0ebe768ebeee-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.161002 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.160981    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/96b558af-774c-4a05-a70f-0ebe768ebeee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.162481 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.162458    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96b558af-774c-4a05-a70f-0ebe768ebeee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.162577 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.162557    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.171083 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.171060    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6mr4\" (UniqueName: \"kubernetes.io/projected/96b558af-774c-4a05-a70f-0ebe768ebeee-kube-api-access-b6mr4\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.171182 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.171129    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnkrg\" (UniqueName: \"kubernetes.io/projected/303a511a-4ac0-4b7d-bc70-e21fde77d433-kube-api-access-nnkrg\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.270438 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.270408    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:11.368567 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.368537    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:11.469574 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.469544    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:11.569353 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.569316    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:11.666124 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.666038    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96b558af-774c-4a05-a70f-0ebe768ebeee-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.666309 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.666143    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-tls\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.668310 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.668278    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/303a511a-4ac0-4b7d-bc70-e21fde77d433-node-exporter-tls\") pod \"node-exporter-4nmrq\" (UID: \"303a511a-4ac0-4b7d-bc70-e21fde77d433\") " pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.668480 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.668457    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96b558af-774c-4a05-a70f-0ebe768ebeee-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-qspqh\" (UID: \"96b558af-774c-4a05-a70f-0ebe768ebeee\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.956360 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.956295    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4nmrq"
Jul 28 20:49:11.963083 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.963066    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"
Jul 28 20:49:11.967690 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:49:11.967648    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303a511a_4ac0_4b7d_bc70_e21fde77d433.slice/crio-fe03351503b940e40b9139e447d2f4c013db2efde6b9d2f3fd9bc94b9841d0c2 WatchSource:0}: Error finding container fe03351503b940e40b9139e447d2f4c013db2efde6b9d2f3fd9bc94b9841d0c2: Status 404 returned error can't find the container with id fe03351503b940e40b9139e447d2f4c013db2efde6b9d2f3fd9bc94b9841d0c2
Jul 28 20:49:11.984199 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:11.984174    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:12.083326 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:12.083287    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:12.092778 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:12.092758    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh"]
Jul 28 20:49:12.095434 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:49:12.095413    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b558af_774c_4a05_a70f_0ebe768ebeee.slice/crio-3ab60e0af6b9ec62fb8a7443d0b8f97af9059e8a7de487f477b2b3b615259f21 WatchSource:0}: Error finding container 3ab60e0af6b9ec62fb8a7443d0b8f97af9059e8a7de487f477b2b3b615259f21: Status 404 returned error can't find the container with id 3ab60e0af6b9ec62fb8a7443d0b8f97af9059e8a7de487f477b2b3b615259f21
Jul 28 20:49:12.411885 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:12.411844    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mrnqh"
Jul 28 20:49:12.450374 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:12.450345    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gprrj" event={"ID":"55e48be1-28c7-409d-baf5-939aeeff7d34","Type":"ContainerStarted","Data":"454e5b29c8bd1c210ad68ac62437a9d2c8f8773f31afdf6a7a2497b83bc2a1b1"}
Jul 28 20:49:12.451420 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:12.451401    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh" event={"ID":"96b558af-774c-4a05-a70f-0ebe768ebeee","Type":"ContainerStarted","Data":"3ab60e0af6b9ec62fb8a7443d0b8f97af9059e8a7de487f477b2b3b615259f21"}
Jul 28 20:49:12.452485 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:12.452463    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4nmrq" event={"ID":"303a511a-4ac0-4b7d-bc70-e21fde77d433","Type":"ContainerStarted","Data":"fe03351503b940e40b9139e447d2f4c013db2efde6b9d2f3fd9bc94b9841d0c2"}
Jul 28 20:49:12.494789 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:12.494749    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gprrj" podStartSLOduration=34.181136331 podStartE2EDuration="47.494736085s" totalImagesPullingTime="13.313599754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:48:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:12.493214128 +0000 UTC m=+113.110796064" watchObservedRunningTime="2026-07-28 20:49:12.494736085 +0000 UTC m=+113.112317986"
Jul 28 20:49:12.628236 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:12.628203    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:12.725180 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:12.725109    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:13.962142 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:13.962060    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:14.020624 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:14.020592    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:14.461827 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:14.461791    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh" event={"ID":"96b558af-774c-4a05-a70f-0ebe768ebeee","Type":"ContainerStarted","Data":"b29e39f62c70f9cdf1e662358404b9bbd45e2a87dc152d1a6dd4c67545a98435"}
Jul 28 20:49:14.462019 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:14.461847    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh" event={"ID":"96b558af-774c-4a05-a70f-0ebe768ebeee","Type":"ContainerStarted","Data":"e4ce30d1da3db2db49974d98f0d0853570076a141594ccd38748529bb3888d27"}
Jul 28 20:49:14.462019 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:14.461891    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh" event={"ID":"96b558af-774c-4a05-a70f-0ebe768ebeee","Type":"ContainerStarted","Data":"d13eed6ca59a37cfd6b1d61cc7de079beda51bcef927d84b1d8ba238b417cd94"}
Jul 28 20:49:14.463376 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:14.463354    2537 generic.go:356] "Generic (PLEG): container finished" podID="303a511a-4ac0-4b7d-bc70-e21fde77d433" containerID="47934c98ae0b8abf92da56657a509f77150c152c87c2084877f6f8b328ec2ec2" exitCode=0
Jul 28 20:49:14.463473 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:14.463429    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4nmrq" event={"ID":"303a511a-4ac0-4b7d-bc70-e21fde77d433","Type":"ContainerDied","Data":"47934c98ae0b8abf92da56657a509f77150c152c87c2084877f6f8b328ec2ec2"}
Jul 28 20:49:14.487054 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:14.486969    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qspqh" podStartSLOduration=2.145834527 podStartE2EDuration="3.486954749s" totalImagesPullingTime="1.341120222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:14.484285213 +0000 UTC m=+115.101867114" watchObservedRunningTime="2026-07-28 20:49:14.486954749 +0000 UTC m=+115.104536684"
Jul 28 20:49:14.547972 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:14.547949    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:14.660664 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:14.660628    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:15.469564 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:15.469524    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4nmrq" event={"ID":"303a511a-4ac0-4b7d-bc70-e21fde77d433","Type":"ContainerStarted","Data":"8d416317eea5a62f554127ee5b5f4239786f54a0d5971486b1a924f465039cd6"}
Jul 28 20:49:15.470033 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:15.469609    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4nmrq" event={"ID":"303a511a-4ac0-4b7d-bc70-e21fde77d433","Type":"ContainerStarted","Data":"948b86e5078b183ced4e03fc2e9e08f02b5b8838c7e036bc4a7095758d8cdda6"}
Jul 28 20:49:15.493816 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:15.493768    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4nmrq" podStartSLOduration=2.952965908 podStartE2EDuration="4.493754022s" totalImagesPullingTime="1.465713928s" totalInitContainerRuntime="75.074186ms" isStatefulPod=true podCreationTimestamp="2026-07-28 20:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:15.491914972 +0000 UTC m=+116.109496873" watchObservedRunningTime="2026-07-28 20:49:15.493754022 +0000 UTC m=+116.111335924"
Jul 28 20:49:15.780805 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:15.780718    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:15.791896 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:15.791872    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc"]
Jul 28 20:49:15.794954 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:15.794938    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc"
Jul 28 20:49:15.797492 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:15.797471    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 20:49:15.797971 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:15.797952    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bcrqw\""
Jul 28 20:49:15.810018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:15.809997    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc"]
Jul 28 20:49:15.872219 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:15.872190    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:15.900058 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:15.900034    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9vbm\" (UniqueName: \"kubernetes.io/projected/3d268dea-f385-4c8d-9a65-ca2c8e9d6ff3-kube-api-access-t9vbm\") pod \"monitoring-plugin-68b7fc9574-4bjnc\" (UID: \"3d268dea-f385-4c8d-9a65-ca2c8e9d6ff3\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc"
Jul 28 20:49:15.900255 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:15.900096    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d268dea-f385-4c8d-9a65-ca2c8e9d6ff3-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-4bjnc\" (UID: \"3d268dea-f385-4c8d-9a65-ca2c8e9d6ff3\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc"
Jul 28 20:49:15.971410 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:15.971383    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:16.000884 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:16.000840    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d268dea-f385-4c8d-9a65-ca2c8e9d6ff3-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-4bjnc\" (UID: \"3d268dea-f385-4c8d-9a65-ca2c8e9d6ff3\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc"
Jul 28 20:49:16.000976 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:16.000952    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9vbm\" (UniqueName: \"kubernetes.io/projected/3d268dea-f385-4c8d-9a65-ca2c8e9d6ff3-kube-api-access-t9vbm\") pod \"monitoring-plugin-68b7fc9574-4bjnc\" (UID: \"3d268dea-f385-4c8d-9a65-ca2c8e9d6ff3\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc"
Jul 28 20:49:16.003273 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:16.003252    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d268dea-f385-4c8d-9a65-ca2c8e9d6ff3-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-4bjnc\" (UID: \"3d268dea-f385-4c8d-9a65-ca2c8e9d6ff3\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc"
Jul 28 20:49:16.010801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:16.010779    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9vbm\" (UniqueName: \"kubernetes.io/projected/3d268dea-f385-4c8d-9a65-ca2c8e9d6ff3-kube-api-access-t9vbm\") pod \"monitoring-plugin-68b7fc9574-4bjnc\" (UID: \"3d268dea-f385-4c8d-9a65-ca2c8e9d6ff3\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc"
Jul 28 20:49:16.103692 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:16.103668    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc"
Jul 28 20:49:16.230920 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:16.230888    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc"]
Jul 28 20:49:16.235619 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:49:16.235587    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d268dea_f385_4c8d_9a65_ca2c8e9d6ff3.slice/crio-a7f7554257de576ecd87493e8a95fdff62eb813598ca6cccf67287bf1a75d8f3 WatchSource:0}: Error finding container a7f7554257de576ecd87493e8a95fdff62eb813598ca6cccf67287bf1a75d8f3: Status 404 returned error can't find the container with id a7f7554257de576ecd87493e8a95fdff62eb813598ca6cccf67287bf1a75d8f3
Jul 28 20:49:16.473673 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:16.473601    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc" event={"ID":"3d268dea-f385-4c8d-9a65-ca2c8e9d6ff3","Type":"ContainerStarted","Data":"a7f7554257de576ecd87493e8a95fdff62eb813598ca6cccf67287bf1a75d8f3"}
Jul 28 20:49:16.556053 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:16.556022    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:16.843435 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:16.843398    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:17.374982 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:17.374954    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:17.926935 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:17.926902    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:18.021730 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:18.021699    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:18.115107 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:18.115076    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:18.482721 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:18.482682    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc" event={"ID":"3d268dea-f385-4c8d-9a65-ca2c8e9d6ff3","Type":"ContainerStarted","Data":"7c47fc68591f0e4c7371d41477ab722253536ccbe2caa8d6e3c9ef6d436a3abc"}
Jul 28 20:49:18.482926 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:18.482903    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc"
Jul 28 20:49:18.487799 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:18.487778    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc"
Jul 28 20:49:18.506864 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:18.506822    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-4bjnc" podStartSLOduration=2.309901142 podStartE2EDuration="3.506809256s" totalImagesPullingTime="1.196908114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:18.505649372 +0000 UTC m=+119.123231274" watchObservedRunningTime="2026-07-28 20:49:18.506809256 +0000 UTC m=+119.124391158"
Jul 28 20:49:21.600509 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:21.600477    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:21.655247 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:21.655218    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:21.956501 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:21.956420    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:22.058049 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:22.058019    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:25.186197 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:25.186163    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:25.285644 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:25.285612    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:27.067945 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:27.067911    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:27.179221 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:27.179190    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:27.866174 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:27.866145    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:27.941384 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:27.941354    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:28.219768 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:28.219695    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:28.320230 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:28.320196    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:28.595942 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:28.595903    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:28.692328 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:28.692296    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:29.603703 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:29.603668    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs\") pod \"network-metrics-daemon-9qhw8\" (UID: \"5e0b9b3f-4257-4183-90eb-1b7102f3606d\") " pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:49:29.606070 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:29.606047    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e0b9b3f-4257-4183-90eb-1b7102f3606d-metrics-certs\") pod \"network-metrics-daemon-9qhw8\" (UID: \"5e0b9b3f-4257-4183-90eb-1b7102f3606d\") " pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:49:29.640694 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:29.640666    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gvjsj\""
Jul 28 20:49:29.648117 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:29.648097    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qhw8"
Jul 28 20:49:29.776020 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:29.775947    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9qhw8"]
Jul 28 20:49:29.778502 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:49:29.778473    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0b9b3f_4257_4183_90eb_1b7102f3606d.slice/crio-b99ce6bd5f059cbdfa7704df6d63cf8bebbd205ef7674e152d152af1d215b789 WatchSource:0}: Error finding container b99ce6bd5f059cbdfa7704df6d63cf8bebbd205ef7674e152d152af1d215b789: Status 404 returned error can't find the container with id b99ce6bd5f059cbdfa7704df6d63cf8bebbd205ef7674e152d152af1d215b789
Jul 28 20:49:30.523781 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:30.523746    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9qhw8" event={"ID":"5e0b9b3f-4257-4183-90eb-1b7102f3606d","Type":"ContainerStarted","Data":"b99ce6bd5f059cbdfa7704df6d63cf8bebbd205ef7674e152d152af1d215b789"}
Jul 28 20:49:30.997351 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:30.997318    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:31.097927 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:31.097895    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:31.531429 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:31.531393    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9qhw8" event={"ID":"5e0b9b3f-4257-4183-90eb-1b7102f3606d","Type":"ContainerStarted","Data":"e1e6d8a66cfaba685cc73c91eeeb9b8db5395fc77a9b7673f7ec2f94bb126d04"}
Jul 28 20:49:31.531618 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:31.531448    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9qhw8" event={"ID":"5e0b9b3f-4257-4183-90eb-1b7102f3606d","Type":"ContainerStarted","Data":"01898980fcd5cc511626a1dc28bf99a6c971424daa6706d846f42cd602a4c08f"}
Jul 28 20:49:31.557968 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:31.557921    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9qhw8" podStartSLOduration=130.771014419 podStartE2EDuration="2m11.557903132s" totalImagesPullingTime="786.888713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:31.557489506 +0000 UTC m=+132.175071411" watchObservedRunningTime="2026-07-28 20:49:31.557903132 +0000 UTC m=+132.175485038"
Jul 28 20:49:31.680619 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:31.680592    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:31.775056 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:31.775028    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:32.286176 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:32.286143    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:32.388474 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:32.388436    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:32.454823 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:32.454776    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:35.547517 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:35.547480    2537 generic.go:356] "Generic (PLEG): container finished" podID="5cd1b546-a068-41fc-9b32-ff5c2295a654" containerID="98936edb01d54d110a5913015b0b0d1aad5ba13d5bb594ccad8e110d91353dfa" exitCode=0
Jul 28 20:49:35.547915 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:35.547534    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mpflg" event={"ID":"5cd1b546-a068-41fc-9b32-ff5c2295a654","Type":"ContainerDied","Data":"98936edb01d54d110a5913015b0b0d1aad5ba13d5bb594ccad8e110d91353dfa"}
Jul 28 20:49:35.547915 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:35.547836    2537 scope.go:117] "RemoveContainer" containerID="98936edb01d54d110a5913015b0b0d1aad5ba13d5bb594ccad8e110d91353dfa"
Jul 28 20:49:35.905622 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:35.905593    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:36.003624 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:36.003592    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:36.067893 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:36.067844    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:36.552429 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:36.552399    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mpflg" event={"ID":"5cd1b546-a068-41fc-9b32-ff5c2295a654","Type":"ContainerStarted","Data":"5be2e6ed93d3f016fe7bff59261cb67452dbec3222b3797f074ca0a3285b056e"}
Jul 28 20:49:38.074409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:38.074376    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:38.139785 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:38.139760    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:43.053299 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:43.053240    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:43.153161 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:43.153116    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:43.575013 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:43.574985    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:43.646075 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:49:43.646043    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:27.225448 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:50:27.225362    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:27.283914 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:50:27.283875    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:38.618546 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:50:38.618506    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:38.761275 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:50:38.761249    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:39.013301 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:50:39.013220    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:39.114460 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:50:39.114428    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:43.568822 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:50:43.568786    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:43.670060 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:50:43.670023    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:43.740563 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:50:43.740530    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:49.382069 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:50:49.382030    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:49.442128 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:50:49.442102    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:49.501606 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:50:49.501573    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:06.229752 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:06.229713    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:06.993356 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:06.993325    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:07.089520 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:07.089493    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:23.437005 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:23.436971    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:26.130876 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:26.130830    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:28.649537 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:28.649501    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:29.437088 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:29.437034    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:32.250282 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:32.250245    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:32.364502 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:32.364474    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:33.894269 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:33.894226    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:33.983872 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:33.983822    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:34.136781 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:34.136746    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:34.243577 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:34.243495    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:36.476126 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:36.476092    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:36.697156 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:36.697118    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:39.069296 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:39.069261    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:39.127162 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:39.127131    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:43.688184 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:43.688105    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:44.513736 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:44.513703    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-cjp7t"]
Jul 28 20:51:44.516669 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:44.516652    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-cjp7t"
Jul 28 20:51:44.519863 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:44.519830    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 20:51:44.520141 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:44.520123    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 20:51:44.521015 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:44.521002    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ghgs5\""
Jul 28 20:51:44.540215 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:44.540195    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-cjp7t"]
Jul 28 20:51:44.625707 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:44.625672    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b0a2a87-ce8b-49d6-a911-7447702f1261-bound-sa-token\") pod \"cert-manager-55b4d954d5-cjp7t\" (UID: \"8b0a2a87-ce8b-49d6-a911-7447702f1261\") " pod="cert-manager/cert-manager-55b4d954d5-cjp7t"
Jul 28 20:51:44.625882 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:44.625729    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdgww\" (UniqueName: \"kubernetes.io/projected/8b0a2a87-ce8b-49d6-a911-7447702f1261-kube-api-access-sdgww\") pod \"cert-manager-55b4d954d5-cjp7t\" (UID: \"8b0a2a87-ce8b-49d6-a911-7447702f1261\") " pod="cert-manager/cert-manager-55b4d954d5-cjp7t"
Jul 28 20:51:44.726255 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:44.726223    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdgww\" (UniqueName: \"kubernetes.io/projected/8b0a2a87-ce8b-49d6-a911-7447702f1261-kube-api-access-sdgww\") pod \"cert-manager-55b4d954d5-cjp7t\" (UID: \"8b0a2a87-ce8b-49d6-a911-7447702f1261\") " pod="cert-manager/cert-manager-55b4d954d5-cjp7t"
Jul 28 20:51:44.726605 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:44.726266    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b0a2a87-ce8b-49d6-a911-7447702f1261-bound-sa-token\") pod \"cert-manager-55b4d954d5-cjp7t\" (UID: \"8b0a2a87-ce8b-49d6-a911-7447702f1261\") " pod="cert-manager/cert-manager-55b4d954d5-cjp7t"
Jul 28 20:51:44.735261 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:44.735234    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdgww\" (UniqueName: \"kubernetes.io/projected/8b0a2a87-ce8b-49d6-a911-7447702f1261-kube-api-access-sdgww\") pod \"cert-manager-55b4d954d5-cjp7t\" (UID: \"8b0a2a87-ce8b-49d6-a911-7447702f1261\") " pod="cert-manager/cert-manager-55b4d954d5-cjp7t"
Jul 28 20:51:44.735472 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:44.735450    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b0a2a87-ce8b-49d6-a911-7447702f1261-bound-sa-token\") pod \"cert-manager-55b4d954d5-cjp7t\" (UID: \"8b0a2a87-ce8b-49d6-a911-7447702f1261\") " pod="cert-manager/cert-manager-55b4d954d5-cjp7t"
Jul 28 20:51:44.825431 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:44.825407    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-cjp7t"
Jul 28 20:51:44.950792 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:44.950750    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-cjp7t"]
Jul 28 20:51:44.953391 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:51:44.953363    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b0a2a87_ce8b_49d6_a911_7447702f1261.slice/crio-d2daba9b44e95333f7998504b3ebacc47988097caa102a011191abed7d1b9291 WatchSource:0}: Error finding container d2daba9b44e95333f7998504b3ebacc47988097caa102a011191abed7d1b9291: Status 404 returned error can't find the container with id d2daba9b44e95333f7998504b3ebacc47988097caa102a011191abed7d1b9291
Jul 28 20:51:44.980912 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:44.980881    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-cjp7t" event={"ID":"8b0a2a87-ce8b-49d6-a911-7447702f1261","Type":"ContainerStarted","Data":"d2daba9b44e95333f7998504b3ebacc47988097caa102a011191abed7d1b9291"}
Jul 28 20:51:45.678296 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:45.678261    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:45.769473 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:45.769418    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:48.998248 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:48.998212    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-cjp7t" event={"ID":"8b0a2a87-ce8b-49d6-a911-7447702f1261","Type":"ContainerStarted","Data":"54c7d253e189d7f17666d05cd304e4ca963dda350d4814862f4adb68bfbfab43"}
Jul 28 20:51:49.017358 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:49.017312    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-cjp7t" podStartSLOduration=1.734871591 podStartE2EDuration="5.017293405s" totalImagesPullingTime="3.282421814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:51:49.017115137 +0000 UTC m=+269.634697039" watchObservedRunningTime="2026-07-28 20:51:49.017293405 +0000 UTC m=+269.634875308"
Jul 28 20:51:52.188563 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:52.188523    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:52.292596 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:52.292561    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:54.235026 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:54.234957    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:54.580523 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:51:54.580496    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:01.951524 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:01.951488    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:02.029594 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:02.029565    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:06.102830 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:06.102798    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:06.203918 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:06.203889    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:06.722716 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:06.722685    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:06.960499 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:06.960464    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:07.061494 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.061413    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:07.148164 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.148134    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"]
Jul 28 20:52:07.154387 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.154367    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.165247 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.165224    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 20:52:07.165401 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.165358    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 20:52:07.165401 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.165364    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 20:52:07.165524 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.165419    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 20:52:07.165524 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.165423    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sbqvl\""
Jul 28 20:52:07.171781 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.171757    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"]
Jul 28 20:52:07.275293 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.275237    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:07.285592 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.285564    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bcfc884a-fe66-4921-9bec-6d8c8723cf4a-manifests\") pod \"opendatahub-operator-controller-manager-6cc774576c-qt8l9\" (UID: \"bcfc884a-fe66-4921-9bec-6d8c8723cf4a\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.285741 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.285600    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bcfc884a-fe66-4921-9bec-6d8c8723cf4a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cc774576c-qt8l9\" (UID: \"bcfc884a-fe66-4921-9bec-6d8c8723cf4a\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.285741 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.285624    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2sd5\" (UniqueName: \"kubernetes.io/projected/bcfc884a-fe66-4921-9bec-6d8c8723cf4a-kube-api-access-q2sd5\") pod \"opendatahub-operator-controller-manager-6cc774576c-qt8l9\" (UID: \"bcfc884a-fe66-4921-9bec-6d8c8723cf4a\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.285741 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.285710    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcfc884a-fe66-4921-9bec-6d8c8723cf4a-tmp\") pod \"opendatahub-operator-controller-manager-6cc774576c-qt8l9\" (UID: \"bcfc884a-fe66-4921-9bec-6d8c8723cf4a\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.285926 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.285771    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcfc884a-fe66-4921-9bec-6d8c8723cf4a-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cc774576c-qt8l9\" (UID: \"bcfc884a-fe66-4921-9bec-6d8c8723cf4a\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.386375 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.386339    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcfc884a-fe66-4921-9bec-6d8c8723cf4a-tmp\") pod \"opendatahub-operator-controller-manager-6cc774576c-qt8l9\" (UID: \"bcfc884a-fe66-4921-9bec-6d8c8723cf4a\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.386542 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.386395    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcfc884a-fe66-4921-9bec-6d8c8723cf4a-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cc774576c-qt8l9\" (UID: \"bcfc884a-fe66-4921-9bec-6d8c8723cf4a\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.386542 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.386443    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bcfc884a-fe66-4921-9bec-6d8c8723cf4a-manifests\") pod \"opendatahub-operator-controller-manager-6cc774576c-qt8l9\" (UID: \"bcfc884a-fe66-4921-9bec-6d8c8723cf4a\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.386542 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.386467    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bcfc884a-fe66-4921-9bec-6d8c8723cf4a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cc774576c-qt8l9\" (UID: \"bcfc884a-fe66-4921-9bec-6d8c8723cf4a\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.386542 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.386494    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2sd5\" (UniqueName: \"kubernetes.io/projected/bcfc884a-fe66-4921-9bec-6d8c8723cf4a-kube-api-access-q2sd5\") pod \"opendatahub-operator-controller-manager-6cc774576c-qt8l9\" (UID: \"bcfc884a-fe66-4921-9bec-6d8c8723cf4a\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.386851 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.386826    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bcfc884a-fe66-4921-9bec-6d8c8723cf4a-manifests\") pod \"opendatahub-operator-controller-manager-6cc774576c-qt8l9\" (UID: \"bcfc884a-fe66-4921-9bec-6d8c8723cf4a\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.388791 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.388762    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcfc884a-fe66-4921-9bec-6d8c8723cf4a-tmp\") pod \"opendatahub-operator-controller-manager-6cc774576c-qt8l9\" (UID: \"bcfc884a-fe66-4921-9bec-6d8c8723cf4a\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.388995 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.388973    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bcfc884a-fe66-4921-9bec-6d8c8723cf4a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cc774576c-qt8l9\" (UID: \"bcfc884a-fe66-4921-9bec-6d8c8723cf4a\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.389071 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.389047    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcfc884a-fe66-4921-9bec-6d8c8723cf4a-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cc774576c-qt8l9\" (UID: \"bcfc884a-fe66-4921-9bec-6d8c8723cf4a\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.416132 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.416107    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2sd5\" (UniqueName: \"kubernetes.io/projected/bcfc884a-fe66-4921-9bec-6d8c8723cf4a-kube-api-access-q2sd5\") pod \"opendatahub-operator-controller-manager-6cc774576c-qt8l9\" (UID: \"bcfc884a-fe66-4921-9bec-6d8c8723cf4a\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.423286 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.423265    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:07.465303 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.465282    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:07.537871 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.537823    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:07.686729 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.686687    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"]
Jul 28 20:52:07.688622 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:52:07.688576    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc884a_fe66_4921_9bec_6d8c8723cf4a.slice/crio-49ade3c43233cd06263a8291a6b010f801fa1798fe605caf1a1fb15f4812294e WatchSource:0}: Error finding container 49ade3c43233cd06263a8291a6b010f801fa1798fe605caf1a1fb15f4812294e: Status 404 returned error can't find the container with id 49ade3c43233cd06263a8291a6b010f801fa1798fe605caf1a1fb15f4812294e
Jul 28 20:52:07.708968 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.708942    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:07.792161 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.792123    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:07.891224 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:07.891193    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:08.060651 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:08.060572    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9" event={"ID":"bcfc884a-fe66-4921-9bec-6d8c8723cf4a","Type":"ContainerStarted","Data":"49ade3c43233cd06263a8291a6b010f801fa1798fe605caf1a1fb15f4812294e"}
Jul 28 20:52:08.562071 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:08.562032    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:08.654054 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:08.654001    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:11.072790 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:11.072746    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9" event={"ID":"bcfc884a-fe66-4921-9bec-6d8c8723cf4a","Type":"ContainerStarted","Data":"cef9b3edaaa690c71e3d6eb73584ce0049018a97d9c6d03d84dc1b54a82a19d8"}
Jul 28 20:52:12.077891 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:12.077771    2537 generic.go:356] "Generic (PLEG): container finished" podID="bcfc884a-fe66-4921-9bec-6d8c8723cf4a" containerID="cef9b3edaaa690c71e3d6eb73584ce0049018a97d9c6d03d84dc1b54a82a19d8" exitCode=0
Jul 28 20:52:12.077891 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:12.077868    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9" event={"ID":"bcfc884a-fe66-4921-9bec-6d8c8723cf4a","Type":"ContainerDied","Data":"cef9b3edaaa690c71e3d6eb73584ce0049018a97d9c6d03d84dc1b54a82a19d8"}
Jul 28 20:52:13.083607 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:13.083574    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9" event={"ID":"bcfc884a-fe66-4921-9bec-6d8c8723cf4a","Type":"ContainerStarted","Data":"6e750cb15b7031e7b290eaef76fee6a808c3698081a0cfb5bdf4c6ed0d64d7a8"}
Jul 28 20:52:13.084029 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:13.083690    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:13.107081 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:13.107031    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9" podStartSLOduration=1.682363735 podStartE2EDuration="6.107015146s" totalImagesPullingTime="3.43600034s" totalInitContainerRuntime="988.651071ms" isStatefulPod=false podCreationTimestamp="2026-07-28 20:52:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:52:13.105922318 +0000 UTC m=+293.723504219" watchObservedRunningTime="2026-07-28 20:52:13.107015146 +0000 UTC m=+293.724597047"
Jul 28 20:52:19.796936 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:19.796899    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 20:52:19.804175 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:19.804146    2537 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 20:52:19.809641 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:19.809620    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 20:52:24.088901 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:24.088845    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-qt8l9"
Jul 28 20:52:26.233523 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:26.233474    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:26.338386 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:26.338349    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:26.763626 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:26.763596    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:26.814772 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:26.814739    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl"]
Jul 28 20:52:26.817967 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:26.817946    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl"
Jul 28 20:52:26.821727 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:26.821706    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 20:52:26.821982 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:26.821964    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hc5bk\""
Jul 28 20:52:26.824453 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:26.824298    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 20:52:26.835157 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:26.835138    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl"]
Jul 28 20:52:26.931575 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:26.931547    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbszv\" (UniqueName: \"kubernetes.io/projected/618aeb6b-d661-425a-b379-04280daef097-kube-api-access-dbszv\") pod \"kube-auth-proxy-68cb4f9b6-pmtxl\" (UID: \"618aeb6b-d661-425a-b379-04280daef097\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl"
Jul 28 20:52:26.931730 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:26.931610    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/618aeb6b-d661-425a-b379-04280daef097-tls-certs\") pod \"kube-auth-proxy-68cb4f9b6-pmtxl\" (UID: \"618aeb6b-d661-425a-b379-04280daef097\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl"
Jul 28 20:52:26.931730 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:26.931635    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/618aeb6b-d661-425a-b379-04280daef097-tmp\") pod \"kube-auth-proxy-68cb4f9b6-pmtxl\" (UID: \"618aeb6b-d661-425a-b379-04280daef097\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl"
Jul 28 20:52:27.032797 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:27.032733    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbszv\" (UniqueName: \"kubernetes.io/projected/618aeb6b-d661-425a-b379-04280daef097-kube-api-access-dbszv\") pod \"kube-auth-proxy-68cb4f9b6-pmtxl\" (UID: \"618aeb6b-d661-425a-b379-04280daef097\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl"
Jul 28 20:52:27.032797 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:27.032775    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/618aeb6b-d661-425a-b379-04280daef097-tls-certs\") pod \"kube-auth-proxy-68cb4f9b6-pmtxl\" (UID: \"618aeb6b-d661-425a-b379-04280daef097\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl"
Jul 28 20:52:27.032797 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:27.032791    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/618aeb6b-d661-425a-b379-04280daef097-tmp\") pod \"kube-auth-proxy-68cb4f9b6-pmtxl\" (UID: \"618aeb6b-d661-425a-b379-04280daef097\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl"
Jul 28 20:52:27.033039 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:52:27.032950    2537 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 28 20:52:27.033039 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:52:27.033018    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/618aeb6b-d661-425a-b379-04280daef097-tls-certs podName:618aeb6b-d661-425a-b379-04280daef097 nodeName:}" failed. No retries permitted until 2026-07-28 20:52:27.533000254 +0000 UTC m=+308.150582134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/618aeb6b-d661-425a-b379-04280daef097-tls-certs") pod "kube-auth-proxy-68cb4f9b6-pmtxl" (UID: "618aeb6b-d661-425a-b379-04280daef097") : secret "kube-auth-proxy-tls" not found
Jul 28 20:52:27.034973 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:27.034958    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/618aeb6b-d661-425a-b379-04280daef097-tmp\") pod \"kube-auth-proxy-68cb4f9b6-pmtxl\" (UID: \"618aeb6b-d661-425a-b379-04280daef097\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl"
Jul 28 20:52:27.043368 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:27.043343    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbszv\" (UniqueName: \"kubernetes.io/projected/618aeb6b-d661-425a-b379-04280daef097-kube-api-access-dbszv\") pod \"kube-auth-proxy-68cb4f9b6-pmtxl\" (UID: \"618aeb6b-d661-425a-b379-04280daef097\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl"
Jul 28 20:52:27.185193 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:27.185161    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:27.287419 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:27.287346    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:27.398406 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:27.398369    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:27.498245 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:27.498210    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:27.536874 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:27.536825    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/618aeb6b-d661-425a-b379-04280daef097-tls-certs\") pod \"kube-auth-proxy-68cb4f9b6-pmtxl\" (UID: \"618aeb6b-d661-425a-b379-04280daef097\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl"
Jul 28 20:52:27.539389 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:27.539329    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/618aeb6b-d661-425a-b379-04280daef097-tls-certs\") pod \"kube-auth-proxy-68cb4f9b6-pmtxl\" (UID: \"618aeb6b-d661-425a-b379-04280daef097\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl"
Jul 28 20:52:27.584169 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:27.584138    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:27.727700 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:27.727667    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl"
Jul 28 20:52:27.872810 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:27.872748    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl"]
Jul 28 20:52:27.875356 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:52:27.875333    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618aeb6b_d661_425a_b379_04280daef097.slice/crio-f9c3765c929ac8d414a52da7f8f4e1c00ba4f24f5cbec0dc82f95a84bd3c1c9f WatchSource:0}: Error finding container f9c3765c929ac8d414a52da7f8f4e1c00ba4f24f5cbec0dc82f95a84bd3c1c9f: Status 404 returned error can't find the container with id f9c3765c929ac8d414a52da7f8f4e1c00ba4f24f5cbec0dc82f95a84bd3c1c9f
Jul 28 20:52:27.877109 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:27.877092    2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:52:28.059181 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.059096    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kxjcl"]
Jul 28 20:52:28.064929 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.064905    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-kxjcl"
Jul 28 20:52:28.068263 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.068244    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 28 20:52:28.077988 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.077964    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kxjcl"]
Jul 28 20:52:28.089210 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.089186    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:28.145673 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.145647    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl" event={"ID":"618aeb6b-d661-425a-b379-04280daef097","Type":"ContainerStarted","Data":"f9c3765c929ac8d414a52da7f8f4e1c00ba4f24f5cbec0dc82f95a84bd3c1c9f"}
Jul 28 20:52:28.242168 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.242139    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b782743-5f3e-4c6b-abfd-3b726e6841cc-tls-certs\") pod \"model-serving-api-54d89bd5fc-kxjcl\" (UID: \"3b782743-5f3e-4c6b-abfd-3b726e6841cc\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxjcl"
Jul 28 20:52:28.242321 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.242197    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc6c5\" (UniqueName: \"kubernetes.io/projected/3b782743-5f3e-4c6b-abfd-3b726e6841cc-kube-api-access-tc6c5\") pod \"model-serving-api-54d89bd5fc-kxjcl\" (UID: \"3b782743-5f3e-4c6b-abfd-3b726e6841cc\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxjcl"
Jul 28 20:52:28.288967 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.288935    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"]
Jul 28 20:52:28.292122 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.292107    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.294205 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.294181    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:28.294793 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.294777    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 28 20:52:28.295044 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.295026    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 28 20:52:28.295205 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.295184    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 20:52:28.295261 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.295216    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-264pc\""
Jul 28 20:52:28.295833 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.295821    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 28 20:52:28.312023 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.309985    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"]
Jul 28 20:52:28.342934 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.342903    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b782743-5f3e-4c6b-abfd-3b726e6841cc-tls-certs\") pod \"model-serving-api-54d89bd5fc-kxjcl\" (UID: \"3b782743-5f3e-4c6b-abfd-3b726e6841cc\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxjcl"
Jul 28 20:52:28.343069 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.343003    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc6c5\" (UniqueName: \"kubernetes.io/projected/3b782743-5f3e-4c6b-abfd-3b726e6841cc-kube-api-access-tc6c5\") pod \"model-serving-api-54d89bd5fc-kxjcl\" (UID: \"3b782743-5f3e-4c6b-abfd-3b726e6841cc\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxjcl"
Jul 28 20:52:28.345945 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.345921    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b782743-5f3e-4c6b-abfd-3b726e6841cc-tls-certs\") pod \"model-serving-api-54d89bd5fc-kxjcl\" (UID: \"3b782743-5f3e-4c6b-abfd-3b726e6841cc\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxjcl"
Jul 28 20:52:28.353282 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.353263    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc6c5\" (UniqueName: \"kubernetes.io/projected/3b782743-5f3e-4c6b-abfd-3b726e6841cc-kube-api-access-tc6c5\") pod \"model-serving-api-54d89bd5fc-kxjcl\" (UID: \"3b782743-5f3e-4c6b-abfd-3b726e6841cc\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxjcl"
Jul 28 20:52:28.366639 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.366607    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:28.374455 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.374436    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-kxjcl"
Jul 28 20:52:28.434435 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.434406    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:28.443423 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.443390    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.443543 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.443453    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.443543 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.443481    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnqfb\" (UniqueName: \"kubernetes.io/projected/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-kube-api-access-pnqfb\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.443543 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.443531    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.443701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.443560    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.443701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.443600    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.443701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.443632    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.535044 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.535015    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kxjcl"]
Jul 28 20:52:28.537165 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:52:28.537127    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b782743_5f3e_4c6b_abfd_3b726e6841cc.slice/crio-ba9a779ca52edb7c11f8d5352af5d4c581453a33c6eeb30448599faa9ab6266c WatchSource:0}: Error finding container ba9a779ca52edb7c11f8d5352af5d4c581453a33c6eeb30448599faa9ab6266c: Status 404 returned error can't find the container with id ba9a779ca52edb7c11f8d5352af5d4c581453a33c6eeb30448599faa9ab6266c
Jul 28 20:52:28.544179 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.544155    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.544276 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.544195    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.544276 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.544237    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.544276 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.544260    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.544435 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.544304    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.544435 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.544356    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.544435 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.544379    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnqfb\" (UniqueName: \"kubernetes.io/projected/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-kube-api-access-pnqfb\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.545217 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.545186    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.546912 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.546889    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.547004 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.546920    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.547137 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.547117    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.547234 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.547220    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.555298 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.555277    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.555777 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.555757    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnqfb\" (UniqueName: \"kubernetes.io/projected/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-kube-api-access-pnqfb\") pod \"istiod-openshift-gateway-7b7995d5cd-l9shq\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.601808 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.601697    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:28.767341 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:28.767311    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"]
Jul 28 20:52:28.770174 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:52:28.770147    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c0e423_2efb_4d4f_b8ef_8ae66bb84acc.slice/crio-fdb65889d2d9e9071f2be30cd7a4d343658648c1282769005b503323e4ac6bd7 WatchSource:0}: Error finding container fdb65889d2d9e9071f2be30cd7a4d343658648c1282769005b503323e4ac6bd7: Status 404 returned error can't find the container with id fdb65889d2d9e9071f2be30cd7a4d343658648c1282769005b503323e4ac6bd7
Jul 28 20:52:29.152635 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:29.152599    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq" event={"ID":"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc","Type":"ContainerStarted","Data":"fdb65889d2d9e9071f2be30cd7a4d343658648c1282769005b503323e4ac6bd7"}
Jul 28 20:52:29.154395 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:29.154332    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-kxjcl" event={"ID":"3b782743-5f3e-4c6b-abfd-3b726e6841cc","Type":"ContainerStarted","Data":"ba9a779ca52edb7c11f8d5352af5d4c581453a33c6eeb30448599faa9ab6266c"}
Jul 28 20:52:29.397475 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:29.397435    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:31.613064 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:31.613027    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 20:52:31.613328 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:31.613099    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 20:52:32.178573 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:32.178525    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq" event={"ID":"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc","Type":"ContainerStarted","Data":"e3beabdb027020842e9aed482a80f468f2ed97b7d19fc00fe6dfa6964a74cf2d"}
Jul 28 20:52:32.178766 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:32.178730    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:32.180231 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:32.180203    2537 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-l9shq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 20:52:32.180340 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:32.180207    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-kxjcl" event={"ID":"3b782743-5f3e-4c6b-abfd-3b726e6841cc","Type":"ContainerStarted","Data":"1cc386339c26d7cfe2c9c8a482ac5bef142a02fbf19f10605f0342403453c3b1"}
Jul 28 20:52:32.180340 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:32.180254    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq" podUID="e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:52:32.180340 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:32.180294    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-kxjcl"
Jul 28 20:52:32.181636 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:32.181608    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl" event={"ID":"618aeb6b-d661-425a-b379-04280daef097","Type":"ContainerStarted","Data":"c45ae5c00600e390bc40f970536fc36b75da337202674c7ccb8346a5a001386d"}
Jul 28 20:52:32.208717 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:32.208668    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq" podStartSLOduration=1.368053657 podStartE2EDuration="4.208651862s" totalImagesPullingTime="2.840598205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:52:32.208509084 +0000 UTC m=+312.826090998" watchObservedRunningTime="2026-07-28 20:52:32.208651862 +0000 UTC m=+312.826233765"
Jul 28 20:52:32.228746 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:32.228703    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-pmtxl" podStartSLOduration=2.554557457 podStartE2EDuration="6.22868833s" totalImagesPullingTime="3.674130873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:52:32.227305298 +0000 UTC m=+312.844887200" watchObservedRunningTime="2026-07-28 20:52:32.22868833 +0000 UTC m=+312.846270235"
Jul 28 20:52:32.246960 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:32.246914    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-kxjcl" podStartSLOduration=1.235041783 podStartE2EDuration="4.246898147s" totalImagesPullingTime="3.011856364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:52:32.246265059 +0000 UTC m=+312.863846963" watchObservedRunningTime="2026-07-28 20:52:32.246898147 +0000 UTC m=+312.864480050"
Jul 28 20:52:32.929571 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:32.929536    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:32.987512 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:32.987476    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:33.186165 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:33.186092    2537 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-l9shq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 20:52:33.186165 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:33.186152    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq" podUID="e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:52:33.623143 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:33.623111    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:33.828477 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:33.828446    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:33.928375 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:33.928302    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:35.527849 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:35.527815    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:35.589142 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:35.589112    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:36.186951 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:36.186919    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:37.523076 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.523045    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:37.643890 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.643840    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj"]
Jul 28 20:52:37.648868 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.648830    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj"
Jul 28 20:52:37.657882 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.657843    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 28 20:52:37.658022 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.658000    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6fqdt\""
Jul 28 20:52:37.685677 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.685617    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:37.700729 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.700703    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg"]
Jul 28 20:52:37.703848 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.703829    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj"]
Jul 28 20:52:37.703992 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.703930    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg"
Jul 28 20:52:37.714914 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.714891    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 28 20:52:37.715170 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.715153    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2jx5l\""
Jul 28 20:52:37.730886 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.730845    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg"]
Jul 28 20:52:37.820401 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.820374    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42a32435-7136-4e80-b9ae-89576182daa4-cert\") pod \"kserve-controller-manager-d6f4b65b8-t7qtj\" (UID: \"42a32435-7136-4e80-b9ae-89576182daa4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj"
Jul 28 20:52:37.820537 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.820416    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q69j\" (UniqueName: \"kubernetes.io/projected/8768f71c-7578-4d35-bab3-66352dea99d8-kube-api-access-7q69j\") pod \"llmisvc-controller-manager-777c6bd7d5-n4vrg\" (UID: \"8768f71c-7578-4d35-bab3-66352dea99d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg"
Jul 28 20:52:37.820537 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.820476    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99lcr\" (UniqueName: \"kubernetes.io/projected/42a32435-7136-4e80-b9ae-89576182daa4-kube-api-access-99lcr\") pod \"kserve-controller-manager-d6f4b65b8-t7qtj\" (UID: \"42a32435-7136-4e80-b9ae-89576182daa4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj"
Jul 28 20:52:37.820537 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.820513    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8768f71c-7578-4d35-bab3-66352dea99d8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-n4vrg\" (UID: \"8768f71c-7578-4d35-bab3-66352dea99d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg"
Jul 28 20:52:37.905336 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.905293    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:37.921772 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.921724    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99lcr\" (UniqueName: \"kubernetes.io/projected/42a32435-7136-4e80-b9ae-89576182daa4-kube-api-access-99lcr\") pod \"kserve-controller-manager-d6f4b65b8-t7qtj\" (UID: \"42a32435-7136-4e80-b9ae-89576182daa4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj"
Jul 28 20:52:37.921772 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.921769    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8768f71c-7578-4d35-bab3-66352dea99d8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-n4vrg\" (UID: \"8768f71c-7578-4d35-bab3-66352dea99d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg"
Jul 28 20:52:37.922062 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.921921    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42a32435-7136-4e80-b9ae-89576182daa4-cert\") pod \"kserve-controller-manager-d6f4b65b8-t7qtj\" (UID: \"42a32435-7136-4e80-b9ae-89576182daa4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj"
Jul 28 20:52:37.922062 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.921988    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q69j\" (UniqueName: \"kubernetes.io/projected/8768f71c-7578-4d35-bab3-66352dea99d8-kube-api-access-7q69j\") pod \"llmisvc-controller-manager-777c6bd7d5-n4vrg\" (UID: \"8768f71c-7578-4d35-bab3-66352dea99d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg"
Jul 28 20:52:37.924239 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.924213    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8768f71c-7578-4d35-bab3-66352dea99d8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-n4vrg\" (UID: \"8768f71c-7578-4d35-bab3-66352dea99d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg"
Jul 28 20:52:37.924592 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.924571    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42a32435-7136-4e80-b9ae-89576182daa4-cert\") pod \"kserve-controller-manager-d6f4b65b8-t7qtj\" (UID: \"42a32435-7136-4e80-b9ae-89576182daa4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj"
Jul 28 20:52:37.931679 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.931653    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99lcr\" (UniqueName: \"kubernetes.io/projected/42a32435-7136-4e80-b9ae-89576182daa4-kube-api-access-99lcr\") pod \"kserve-controller-manager-d6f4b65b8-t7qtj\" (UID: \"42a32435-7136-4e80-b9ae-89576182daa4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj"
Jul 28 20:52:37.932421 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.932393    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q69j\" (UniqueName: \"kubernetes.io/projected/8768f71c-7578-4d35-bab3-66352dea99d8-kube-api-access-7q69j\") pod \"llmisvc-controller-manager-777c6bd7d5-n4vrg\" (UID: \"8768f71c-7578-4d35-bab3-66352dea99d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg"
Jul 28 20:52:37.959121 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.959097    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj"
Jul 28 20:52:37.962492 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:37.962469    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:38.014469 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:38.014434    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg"
Jul 28 20:52:38.109656 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:38.109519    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj"]
Jul 28 20:52:38.112510 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:52:38.112469    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a32435_7136_4e80_b9ae_89576182daa4.slice/crio-2fe7e83ffb1544bfb971b7eac4ee0c63756763d869f2689d0945be53ec108a4e WatchSource:0}: Error finding container 2fe7e83ffb1544bfb971b7eac4ee0c63756763d869f2689d0945be53ec108a4e: Status 404 returned error can't find the container with id 2fe7e83ffb1544bfb971b7eac4ee0c63756763d869f2689d0945be53ec108a4e
Jul 28 20:52:38.161445 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:38.161420    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg"]
Jul 28 20:52:38.164502 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:52:38.164472    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8768f71c_7578_4d35_bab3_66352dea99d8.slice/crio-02706999148b0ffa8b60737d85697a36cedab1431123c718d26a37c536c711ed WatchSource:0}: Error finding container 02706999148b0ffa8b60737d85697a36cedab1431123c718d26a37c536c711ed: Status 404 returned error can't find the container with id 02706999148b0ffa8b60737d85697a36cedab1431123c718d26a37c536c711ed
Jul 28 20:52:38.206445 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:38.206416    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg" event={"ID":"8768f71c-7578-4d35-bab3-66352dea99d8","Type":"ContainerStarted","Data":"02706999148b0ffa8b60737d85697a36cedab1431123c718d26a37c536c711ed"}
Jul 28 20:52:38.207500 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:38.207477    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj" event={"ID":"42a32435-7136-4e80-b9ae-89576182daa4","Type":"ContainerStarted","Data":"2fe7e83ffb1544bfb971b7eac4ee0c63756763d869f2689d0945be53ec108a4e"}
Jul 28 20:52:38.905525 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:38.905495    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:39.006216 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:39.006186    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:39.107567 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:39.107490    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:40.977386 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:40.977349    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"]
Jul 28 20:52:40.977817 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:40.977568    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq" podUID="e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc" containerName="discovery" containerID="cri-o://e3beabdb027020842e9aed482a80f468f2ed97b7d19fc00fe6dfa6964a74cf2d" gracePeriod=30
Jul 28 20:52:41.463227 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.463203    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:41.551486 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.551454    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-token\") pod \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") "
Jul 28 20:52:41.551486 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.551485    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnqfb\" (UniqueName: \"kubernetes.io/projected/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-kube-api-access-pnqfb\") pod \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") "
Jul 28 20:52:41.551663 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.551510    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-cacerts\") pod \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") "
Jul 28 20:52:41.551663 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.551531    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-kubeconfig\") pod \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") "
Jul 28 20:52:41.551663 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.551559    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-csr-ca-configmap\") pod \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") "
Jul 28 20:52:41.551663 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.551620    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-csr-dns-cert\") pod \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") "
Jul 28 20:52:41.552050 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.552008    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc" (UID: "e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:52:41.554080 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.553963    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-cacerts" (OuterVolumeSpecName: "cacerts") pod "e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc" (UID: "e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:52:41.554080 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.554058    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc" (UID: "e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:52:41.554223 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.554120    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc" (UID: "e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:52:41.554293 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.554247    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-kube-api-access-pnqfb" (OuterVolumeSpecName: "kube-api-access-pnqfb") pod "e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc" (UID: "e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc"). InnerVolumeSpecName "kube-api-access-pnqfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:52:41.554293 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.554278    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc" (UID: "e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:52:41.652078 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.652045    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-local-certs\") pod \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\" (UID: \"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc\") "
Jul 28 20:52:41.652235 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.652208    2537 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-token\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:52:41.652235 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.652219    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnqfb\" (UniqueName: \"kubernetes.io/projected/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-kube-api-access-pnqfb\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:52:41.652235 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.652228    2537 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-cacerts\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:52:41.652346 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.652237    2537 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-kubeconfig\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:52:41.652346 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.652246    2537 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-csr-ca-configmap\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:52:41.652346 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.652254    2537 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-istio-csr-dns-cert\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:52:41.653911 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.653883    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-local-certs" (OuterVolumeSpecName: "local-certs") pod "e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc" (UID: "e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:52:41.752616 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:41.752580    2537 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc-local-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:52:42.229719 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:42.229682    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj" event={"ID":"42a32435-7136-4e80-b9ae-89576182daa4","Type":"ContainerStarted","Data":"05ec55d0c3acee5ae96e2a62d1dfab3f90b1a75d302f0280e33f68d19414c793"}
Jul 28 20:52:42.230185 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:42.229842    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj"
Jul 28 20:52:42.230939 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:42.230915    2537 generic.go:356] "Generic (PLEG): container finished" podID="e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc" containerID="e3beabdb027020842e9aed482a80f468f2ed97b7d19fc00fe6dfa6964a74cf2d" exitCode=0
Jul 28 20:52:42.231048 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:42.230951    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"
Jul 28 20:52:42.231048 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:42.230992    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq" event={"ID":"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc","Type":"ContainerDied","Data":"e3beabdb027020842e9aed482a80f468f2ed97b7d19fc00fe6dfa6964a74cf2d"}
Jul 28 20:52:42.231048 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:42.231030    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq" event={"ID":"e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc","Type":"ContainerDied","Data":"fdb65889d2d9e9071f2be30cd7a4d343658648c1282769005b503323e4ac6bd7"}
Jul 28 20:52:42.231207 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:42.231054    2537 scope.go:117] "RemoveContainer" containerID="e3beabdb027020842e9aed482a80f468f2ed97b7d19fc00fe6dfa6964a74cf2d"
Jul 28 20:52:42.232669 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:42.232648    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg" event={"ID":"8768f71c-7578-4d35-bab3-66352dea99d8","Type":"ContainerStarted","Data":"a68f485152ba60c5b6efdead64f717858b269282feaea369b6d9bc1bff2caf57"}
Jul 28 20:52:42.232876 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:42.232823    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg"
Jul 28 20:52:42.239277 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:42.239256    2537 scope.go:117] "RemoveContainer" containerID="e3beabdb027020842e9aed482a80f468f2ed97b7d19fc00fe6dfa6964a74cf2d"
Jul 28 20:52:42.239524 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:52:42.239505    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3beabdb027020842e9aed482a80f468f2ed97b7d19fc00fe6dfa6964a74cf2d\": container with ID starting with e3beabdb027020842e9aed482a80f468f2ed97b7d19fc00fe6dfa6964a74cf2d not found: ID does not exist" containerID="e3beabdb027020842e9aed482a80f468f2ed97b7d19fc00fe6dfa6964a74cf2d"
Jul 28 20:52:42.239577 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:42.239531    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3beabdb027020842e9aed482a80f468f2ed97b7d19fc00fe6dfa6964a74cf2d"} err="failed to get container status \"e3beabdb027020842e9aed482a80f468f2ed97b7d19fc00fe6dfa6964a74cf2d\": rpc error: code = NotFound desc = could not find container \"e3beabdb027020842e9aed482a80f468f2ed97b7d19fc00fe6dfa6964a74cf2d\": container with ID starting with e3beabdb027020842e9aed482a80f468f2ed97b7d19fc00fe6dfa6964a74cf2d not found: ID does not exist"
Jul 28 20:52:42.247516 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:42.247475    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj" podStartSLOduration=2.022406143 podStartE2EDuration="5.247464084s" totalImagesPullingTime="3.225057941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:52:42.246188311 +0000 UTC m=+322.863770213" watchObservedRunningTime="2026-07-28 20:52:42.247464084 +0000 UTC m=+322.865046018"
Jul 28 20:52:42.261896 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:42.261875    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"]
Jul 28 20:52:42.266136 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:42.266115    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9shq"]
Jul 28 20:52:42.294563 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:42.294486    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg" podStartSLOduration=2.117562882 podStartE2EDuration="5.294476229s" totalImagesPullingTime="3.176913347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:52:42.293991224 +0000 UTC m=+322.911573127" watchObservedRunningTime="2026-07-28 20:52:42.294476229 +0000 UTC m=+322.912058169"
Jul 28 20:52:43.189233 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:43.189203    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-kxjcl"
Jul 28 20:52:44.020371 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:52:44.020339    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc" path="/var/lib/kubelet/pods/e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc/volumes"
Jul 28 20:53:13.240237 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:13.240153    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n4vrg"
Jul 28 20:53:13.243402 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:13.243381    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7qtj"
Jul 28 20:53:27.576841 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:27.576807    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:27.710734 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:27.710699    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ptv6b"]
Jul 28 20:53:27.711023 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:27.711010    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc" containerName="discovery"
Jul 28 20:53:27.711023 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:27.711024    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc" containerName="discovery"
Jul 28 20:53:27.711115 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:27.711088    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9c0e423-2efb-4d4f-b8ef-8ae66bb84acc" containerName="discovery"
Jul 28 20:53:27.717286 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:27.717259    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ptv6b"
Jul 28 20:53:27.720343 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:27.720322    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 20:53:27.721424 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:27.721405    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-jvb4f\""
Jul 28 20:53:27.721525 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:27.721409    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 20:53:27.729073 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:27.729051    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ptv6b"]
Jul 28 20:53:27.774218 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:27.774191    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbzw8\" (UniqueName: \"kubernetes.io/projected/89172e5e-db33-49e3-a084-97143083b178-kube-api-access-qbzw8\") pod \"kuadrant-operator-catalog-ptv6b\" (UID: \"89172e5e-db33-49e3-a084-97143083b178\") " pod="kuadrant-system/kuadrant-operator-catalog-ptv6b"
Jul 28 20:53:27.875380 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:27.875298    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbzw8\" (UniqueName: \"kubernetes.io/projected/89172e5e-db33-49e3-a084-97143083b178-kube-api-access-qbzw8\") pod \"kuadrant-operator-catalog-ptv6b\" (UID: \"89172e5e-db33-49e3-a084-97143083b178\") " pod="kuadrant-system/kuadrant-operator-catalog-ptv6b"
Jul 28 20:53:27.884097 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:27.884072    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbzw8\" (UniqueName: \"kubernetes.io/projected/89172e5e-db33-49e3-a084-97143083b178-kube-api-access-qbzw8\") pod \"kuadrant-operator-catalog-ptv6b\" (UID: \"89172e5e-db33-49e3-a084-97143083b178\") " pod="kuadrant-system/kuadrant-operator-catalog-ptv6b"
Jul 28 20:53:28.026185 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:28.026156    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ptv6b"
Jul 28 20:53:28.074501 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:28.074422    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ptv6b"]
Jul 28 20:53:28.156069 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:28.156020    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ptv6b"]
Jul 28 20:53:28.158572 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:53:28.158533    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89172e5e_db33_49e3_a084_97143083b178.slice/crio-cc567bb61e12ff22f46d7099becda0f23d6a431f82d6a859e5f03df4c2878451 WatchSource:0}: Error finding container cc567bb61e12ff22f46d7099becda0f23d6a431f82d6a859e5f03df4c2878451: Status 404 returned error can't find the container with id cc567bb61e12ff22f46d7099becda0f23d6a431f82d6a859e5f03df4c2878451
Jul 28 20:53:28.280546 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:28.280507    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-82mc7"]
Jul 28 20:53:28.284783 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:28.284760    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-82mc7"
Jul 28 20:53:28.293464 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:28.293442    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-82mc7"]
Jul 28 20:53:28.378243 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:28.378216    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfslb\" (UniqueName: \"kubernetes.io/projected/c39ea625-8476-4119-8960-6cda4db3bce2-kube-api-access-vfslb\") pod \"kuadrant-operator-catalog-82mc7\" (UID: \"c39ea625-8476-4119-8960-6cda4db3bce2\") " pod="kuadrant-system/kuadrant-operator-catalog-82mc7"
Jul 28 20:53:28.429981 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:28.429916    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ptv6b" event={"ID":"89172e5e-db33-49e3-a084-97143083b178","Type":"ContainerStarted","Data":"cc567bb61e12ff22f46d7099becda0f23d6a431f82d6a859e5f03df4c2878451"}
Jul 28 20:53:28.479329 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:28.479300    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfslb\" (UniqueName: \"kubernetes.io/projected/c39ea625-8476-4119-8960-6cda4db3bce2-kube-api-access-vfslb\") pod \"kuadrant-operator-catalog-82mc7\" (UID: \"c39ea625-8476-4119-8960-6cda4db3bce2\") " pod="kuadrant-system/kuadrant-operator-catalog-82mc7"
Jul 28 20:53:28.489513 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:28.489487    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfslb\" (UniqueName: \"kubernetes.io/projected/c39ea625-8476-4119-8960-6cda4db3bce2-kube-api-access-vfslb\") pod \"kuadrant-operator-catalog-82mc7\" (UID: \"c39ea625-8476-4119-8960-6cda4db3bce2\") " pod="kuadrant-system/kuadrant-operator-catalog-82mc7"
Jul 28 20:53:28.594694 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:28.594661    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-82mc7"
Jul 28 20:53:28.764521 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:28.764486    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-82mc7"]
Jul 28 20:53:28.842371 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:53:28.842339    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39ea625_8476_4119_8960_6cda4db3bce2.slice/crio-3f9011d042414175b307c837ca2c8f045fe91b12057605dfd065eec5d4a3de8a WatchSource:0}: Error finding container 3f9011d042414175b307c837ca2c8f045fe91b12057605dfd065eec5d4a3de8a: Status 404 returned error can't find the container with id 3f9011d042414175b307c837ca2c8f045fe91b12057605dfd065eec5d4a3de8a
Jul 28 20:53:29.176958 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:29.176875    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:29.438570 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:29.437494    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-82mc7" event={"ID":"c39ea625-8476-4119-8960-6cda4db3bce2","Type":"ContainerStarted","Data":"3f9011d042414175b307c837ca2c8f045fe91b12057605dfd065eec5d4a3de8a"}
Jul 28 20:53:30.442572 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:30.442481    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ptv6b" event={"ID":"89172e5e-db33-49e3-a084-97143083b178","Type":"ContainerStarted","Data":"0123d0035c3b6dc190a0e986aac3bca2d29172b848e54d6578dc28f90ce83ce5"}
Jul 28 20:53:30.442572 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:30.442526    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-ptv6b" podUID="89172e5e-db33-49e3-a084-97143083b178" containerName="registry-server" containerID="cri-o://0123d0035c3b6dc190a0e986aac3bca2d29172b848e54d6578dc28f90ce83ce5" gracePeriod=2
Jul 28 20:53:30.444029 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:30.444002    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-82mc7" event={"ID":"c39ea625-8476-4119-8960-6cda4db3bce2","Type":"ContainerStarted","Data":"4f1ff43e03bf893bc0b102e7e444425b2072089af2b0724bee3bb08727abea7e"}
Jul 28 20:53:30.461351 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:30.461315    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-ptv6b" podStartSLOduration=1.4853058479999999 podStartE2EDuration="3.461302302s" totalImagesPullingTime="1.975996454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:53:30.460192322 +0000 UTC m=+371.077774225" watchObservedRunningTime="2026-07-28 20:53:30.461302302 +0000 UTC m=+371.078884203"
Jul 28 20:53:30.481593 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:30.481548    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-82mc7" podStartSLOduration=1.186442397 podStartE2EDuration="2.481534219s" totalImagesPullingTime="1.295091822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:53:30.479368428 +0000 UTC m=+371.096950330" watchObservedRunningTime="2026-07-28 20:53:30.481534219 +0000 UTC m=+371.099116123"
Jul 28 20:53:30.683942 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:30.683919    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ptv6b"
Jul 28 20:53:30.798062 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:30.797984    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbzw8\" (UniqueName: \"kubernetes.io/projected/89172e5e-db33-49e3-a084-97143083b178-kube-api-access-qbzw8\") pod \"89172e5e-db33-49e3-a084-97143083b178\" (UID: \"89172e5e-db33-49e3-a084-97143083b178\") "
Jul 28 20:53:30.800043 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:30.800022    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89172e5e-db33-49e3-a084-97143083b178-kube-api-access-qbzw8" (OuterVolumeSpecName: "kube-api-access-qbzw8") pod "89172e5e-db33-49e3-a084-97143083b178" (UID: "89172e5e-db33-49e3-a084-97143083b178"). InnerVolumeSpecName "kube-api-access-qbzw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:53:30.898597 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:30.898567    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbzw8\" (UniqueName: \"kubernetes.io/projected/89172e5e-db33-49e3-a084-97143083b178-kube-api-access-qbzw8\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:53:31.449632 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:31.449543    2537 generic.go:356] "Generic (PLEG): container finished" podID="89172e5e-db33-49e3-a084-97143083b178" containerID="0123d0035c3b6dc190a0e986aac3bca2d29172b848e54d6578dc28f90ce83ce5" exitCode=0
Jul 28 20:53:31.449632 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:31.449603    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ptv6b"
Jul 28 20:53:31.450081 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:31.449632    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ptv6b" event={"ID":"89172e5e-db33-49e3-a084-97143083b178","Type":"ContainerDied","Data":"0123d0035c3b6dc190a0e986aac3bca2d29172b848e54d6578dc28f90ce83ce5"}
Jul 28 20:53:31.450081 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:31.449693    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ptv6b" event={"ID":"89172e5e-db33-49e3-a084-97143083b178","Type":"ContainerDied","Data":"cc567bb61e12ff22f46d7099becda0f23d6a431f82d6a859e5f03df4c2878451"}
Jul 28 20:53:31.450081 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:31.449715    2537 scope.go:117] "RemoveContainer" containerID="0123d0035c3b6dc190a0e986aac3bca2d29172b848e54d6578dc28f90ce83ce5"
Jul 28 20:53:31.458738 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:31.458721    2537 scope.go:117] "RemoveContainer" containerID="0123d0035c3b6dc190a0e986aac3bca2d29172b848e54d6578dc28f90ce83ce5"
Jul 28 20:53:31.458977 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:53:31.458959    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0123d0035c3b6dc190a0e986aac3bca2d29172b848e54d6578dc28f90ce83ce5\": container with ID starting with 0123d0035c3b6dc190a0e986aac3bca2d29172b848e54d6578dc28f90ce83ce5 not found: ID does not exist" containerID="0123d0035c3b6dc190a0e986aac3bca2d29172b848e54d6578dc28f90ce83ce5"
Jul 28 20:53:31.459028 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:31.458985    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0123d0035c3b6dc190a0e986aac3bca2d29172b848e54d6578dc28f90ce83ce5"} err="failed to get container status \"0123d0035c3b6dc190a0e986aac3bca2d29172b848e54d6578dc28f90ce83ce5\": rpc error: code = NotFound desc = could not find container \"0123d0035c3b6dc190a0e986aac3bca2d29172b848e54d6578dc28f90ce83ce5\": container with ID starting with 0123d0035c3b6dc190a0e986aac3bca2d29172b848e54d6578dc28f90ce83ce5 not found: ID does not exist"
Jul 28 20:53:31.474978 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:31.474954    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ptv6b"]
Jul 28 20:53:31.479851 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:31.479833    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ptv6b"]
Jul 28 20:53:32.020132 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:32.020102    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89172e5e-db33-49e3-a084-97143083b178" path="/var/lib/kubelet/pods/89172e5e-db33-49e3-a084-97143083b178/volumes"
Jul 28 20:53:38.595292 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:38.595242    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-82mc7"
Jul 28 20:53:38.595688 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:38.595306    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-82mc7"
Jul 28 20:53:38.616227 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:38.616203    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-82mc7"
Jul 28 20:53:39.503726 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:39.503698    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-82mc7"
Jul 28 20:53:51.750906 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:51.750867    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:54.152399 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:54.152362    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"]
Jul 28 20:53:54.152889 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:54.152802    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89172e5e-db33-49e3-a084-97143083b178" containerName="registry-server"
Jul 28 20:53:54.152889 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:54.152820    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="89172e5e-db33-49e3-a084-97143083b178" containerName="registry-server"
Jul 28 20:53:54.153021 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:54.152916    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="89172e5e-db33-49e3-a084-97143083b178" containerName="registry-server"
Jul 28 20:53:54.155931 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:54.155911    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:53:54.158525 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:54.158507    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-d8p8s\""
Jul 28 20:53:54.164799 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:54.164768    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"]
Jul 28 20:53:54.265806 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:54.265771    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx7k7\" (UniqueName: \"kubernetes.io/projected/b1bed668-540f-425d-bab4-ef1d3f2f5a19-kube-api-access-bx7k7\") pod \"limitador-operator-controller-manager-859d7b78bd-cn5fj\" (UID: \"b1bed668-540f-425d-bab4-ef1d3f2f5a19\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:53:54.366699 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:54.366665    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx7k7\" (UniqueName: \"kubernetes.io/projected/b1bed668-540f-425d-bab4-ef1d3f2f5a19-kube-api-access-bx7k7\") pod \"limitador-operator-controller-manager-859d7b78bd-cn5fj\" (UID: \"b1bed668-540f-425d-bab4-ef1d3f2f5a19\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:53:54.377059 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:54.377035    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx7k7\" (UniqueName: \"kubernetes.io/projected/b1bed668-540f-425d-bab4-ef1d3f2f5a19-kube-api-access-bx7k7\") pod \"limitador-operator-controller-manager-859d7b78bd-cn5fj\" (UID: \"b1bed668-540f-425d-bab4-ef1d3f2f5a19\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:53:54.466295 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:54.466227    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:53:54.593673 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:54.593648    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"]
Jul 28 20:53:54.595933 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:53:54.595905    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bed668_540f_425d_bab4_ef1d3f2f5a19.slice/crio-23d19e8759be9a0be8e844879f0accfac7b9be336e2bb7ecdb871e785a1ad0de WatchSource:0}: Error finding container 23d19e8759be9a0be8e844879f0accfac7b9be336e2bb7ecdb871e785a1ad0de: Status 404 returned error can't find the container with id 23d19e8759be9a0be8e844879f0accfac7b9be336e2bb7ecdb871e785a1ad0de
Jul 28 20:53:55.179012 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:55.178980    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:55.351693 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:55.351662    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:55.547513 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:55.547434    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj" event={"ID":"b1bed668-540f-425d-bab4-ef1d3f2f5a19","Type":"ContainerStarted","Data":"23d19e8759be9a0be8e844879f0accfac7b9be336e2bb7ecdb871e785a1ad0de"}
Jul 28 20:53:57.559273 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:57.559239    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj" event={"ID":"b1bed668-540f-425d-bab4-ef1d3f2f5a19","Type":"ContainerStarted","Data":"40db68e8a930540cdbbe6ce3320afc1efee7423c71a27b6a55cafbffff545a99"}
Jul 28 20:53:57.559682 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:57.559305    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:53:57.580700 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:57.580651    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj" podStartSLOduration=1.5721438490000001 podStartE2EDuration="3.580636988s" totalImagesPullingTime="2.008493139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:53:57.579455982 +0000 UTC m=+398.197037884" watchObservedRunningTime="2026-07-28 20:53:57.580636988 +0000 UTC m=+398.198218890"
Jul 28 20:53:59.450738 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:59.450707    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"]
Jul 28 20:53:59.453813 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:59.453796    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:53:59.457259 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:59.457238    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ngpxq\""
Jul 28 20:53:59.469018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:59.468997    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"]
Jul 28 20:53:59.607160 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:59.607124    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1e30820-1925-4fa0-b25a-6b52485ea72b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" (UID: \"e1e30820-1925-4fa0-b25a-6b52485ea72b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:53:59.607383 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:59.607174    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcpkr\" (UniqueName: \"kubernetes.io/projected/e1e30820-1925-4fa0-b25a-6b52485ea72b-kube-api-access-jcpkr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" (UID: \"e1e30820-1925-4fa0-b25a-6b52485ea72b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:53:59.707913 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:59.707818    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1e30820-1925-4fa0-b25a-6b52485ea72b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" (UID: \"e1e30820-1925-4fa0-b25a-6b52485ea72b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:53:59.707913 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:59.707880    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcpkr\" (UniqueName: \"kubernetes.io/projected/e1e30820-1925-4fa0-b25a-6b52485ea72b-kube-api-access-jcpkr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" (UID: \"e1e30820-1925-4fa0-b25a-6b52485ea72b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:53:59.708189 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:59.708168    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1e30820-1925-4fa0-b25a-6b52485ea72b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" (UID: \"e1e30820-1925-4fa0-b25a-6b52485ea72b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:53:59.709794 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:59.709768    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:59.716785 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:59.716756    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcpkr\" (UniqueName: \"kubernetes.io/projected/e1e30820-1925-4fa0-b25a-6b52485ea72b-kube-api-access-jcpkr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" (UID: \"e1e30820-1925-4fa0-b25a-6b52485ea72b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:53:59.763275 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:59.763250    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:53:59.808560 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:59.808522    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:59.896086 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:59.896052    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"]
Jul 28 20:53:59.899851 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:53:59.899816    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e30820_1925_4fa0_b25a_6b52485ea72b.slice/crio-aed9ec7aa91da631734bf805ea682b7f496fff02678200df6643e5eb95b7919b WatchSource:0}: Error finding container aed9ec7aa91da631734bf805ea682b7f496fff02678200df6643e5eb95b7919b: Status 404 returned error can't find the container with id aed9ec7aa91da631734bf805ea682b7f496fff02678200df6643e5eb95b7919b
Jul 28 20:53:59.913904 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:53:59.913881    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:00.524746 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:00.524716    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:00.577479 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:00.577441    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl" event={"ID":"e1e30820-1925-4fa0-b25a-6b52485ea72b","Type":"ContainerStarted","Data":"aed9ec7aa91da631734bf805ea682b7f496fff02678200df6643e5eb95b7919b"}
Jul 28 20:54:01.662184 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:01.662140    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:01.761208 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:01.761173    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:02.153491 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:02.153454    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:02.548551 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:02.548468    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-82sks"]
Jul 28 20:54:02.556266 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:02.556246    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-82sks"
Jul 28 20:54:02.559621 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:02.559589    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gwrkb\""
Jul 28 20:54:02.559887 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:02.559867    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-82sks"]
Jul 28 20:54:02.630480 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:02.630451    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngfxk\" (UniqueName: \"kubernetes.io/projected/959b3684-e080-494d-aefd-34c648b9c66e-kube-api-access-ngfxk\") pod \"authorino-operator-75cc8d5cb7-82sks\" (UID: \"959b3684-e080-494d-aefd-34c648b9c66e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-82sks"
Jul 28 20:54:02.731139 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:02.731107    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfxk\" (UniqueName: \"kubernetes.io/projected/959b3684-e080-494d-aefd-34c648b9c66e-kube-api-access-ngfxk\") pod \"authorino-operator-75cc8d5cb7-82sks\" (UID: \"959b3684-e080-494d-aefd-34c648b9c66e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-82sks"
Jul 28 20:54:02.743552 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:02.743526    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngfxk\" (UniqueName: \"kubernetes.io/projected/959b3684-e080-494d-aefd-34c648b9c66e-kube-api-access-ngfxk\") pod \"authorino-operator-75cc8d5cb7-82sks\" (UID: \"959b3684-e080-494d-aefd-34c648b9c66e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-82sks"
Jul 28 20:54:02.868486 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:02.868452    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-82sks"
Jul 28 20:54:03.748537 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:03.748500    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:04.021328 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:04.021303    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-82sks"]
Jul 28 20:54:04.023030 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:54:04.023006    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959b3684_e080_494d_aefd_34c648b9c66e.slice/crio-c27122e0a9d58e763e6f186b076c04f4e856c8249ecc380330bf4112d9cb3810 WatchSource:0}: Error finding container c27122e0a9d58e763e6f186b076c04f4e856c8249ecc380330bf4112d9cb3810: Status 404 returned error can't find the container with id c27122e0a9d58e763e6f186b076c04f4e856c8249ecc380330bf4112d9cb3810
Jul 28 20:54:04.605048 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:04.604971    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-82sks" event={"ID":"959b3684-e080-494d-aefd-34c648b9c66e","Type":"ContainerStarted","Data":"c27122e0a9d58e763e6f186b076c04f4e856c8249ecc380330bf4112d9cb3810"}
Jul 28 20:54:04.612244 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:04.609929    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl" event={"ID":"e1e30820-1925-4fa0-b25a-6b52485ea72b","Type":"ContainerStarted","Data":"81c98929e1a58096bcef9fd221202af4b3f8e065de59052eb08f710db80bae38"}
Jul 28 20:54:04.612244 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:04.610099    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:54:04.667990 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:04.667961    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:04.774200 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:04.774147    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:05.915507 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:05.915445    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl" podStartSLOduration=2.8744278640000003 podStartE2EDuration="6.915426431s" totalImagesPullingTime="4.040998567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:54:04.643391958 +0000 UTC m=+405.260973861" watchObservedRunningTime="2026-07-28 20:54:05.915426431 +0000 UTC m=+406.533008332"
Jul 28 20:54:05.916440 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:05.916414    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-9pxl8"]
Jul 28 20:54:05.920451 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:05.920433    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9pxl8"
Jul 28 20:54:05.923701 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:05.923681    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 28 20:54:05.923801 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:05.923722    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-b2tmr\""
Jul 28 20:54:05.929361 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:05.929336    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-9pxl8"]
Jul 28 20:54:06.061512 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:06.061475    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlgbs\" (UniqueName: \"kubernetes.io/projected/968405d0-4f21-454c-9c39-2367334686a8-kube-api-access-zlgbs\") pod \"dns-operator-controller-manager-864dd78c4c-9pxl8\" (UID: \"968405d0-4f21-454c-9c39-2367334686a8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9pxl8"
Jul 28 20:54:06.162018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:06.161992    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlgbs\" (UniqueName: \"kubernetes.io/projected/968405d0-4f21-454c-9c39-2367334686a8-kube-api-access-zlgbs\") pod \"dns-operator-controller-manager-864dd78c4c-9pxl8\" (UID: \"968405d0-4f21-454c-9c39-2367334686a8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9pxl8"
Jul 28 20:54:06.170822 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:06.170775    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlgbs\" (UniqueName: \"kubernetes.io/projected/968405d0-4f21-454c-9c39-2367334686a8-kube-api-access-zlgbs\") pod \"dns-operator-controller-manager-864dd78c4c-9pxl8\" (UID: \"968405d0-4f21-454c-9c39-2367334686a8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9pxl8"
Jul 28 20:54:06.233612 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:06.233583    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9pxl8"
Jul 28 20:54:06.371620 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:06.371595    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-9pxl8"]
Jul 28 20:54:06.373995 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:54:06.373969    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968405d0_4f21_454c_9c39_2367334686a8.slice/crio-4b60b94263609bbc70be249dc9d5e4de7efefa42f1969a692ae52acda18d2981 WatchSource:0}: Error finding container 4b60b94263609bbc70be249dc9d5e4de7efefa42f1969a692ae52acda18d2981: Status 404 returned error can't find the container with id 4b60b94263609bbc70be249dc9d5e4de7efefa42f1969a692ae52acda18d2981
Jul 28 20:54:06.554090 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:06.554020    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:06.620358 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:06.620321    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-82sks" event={"ID":"959b3684-e080-494d-aefd-34c648b9c66e","Type":"ContainerStarted","Data":"37a7b6ddfb77db1fae1f38ae614bdcf305f91c0491bd6c7f506bc28264a8a04e"}
Jul 28 20:54:06.620498 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:06.620412    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-82sks"
Jul 28 20:54:06.621622 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:06.621601    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9pxl8" event={"ID":"968405d0-4f21-454c-9c39-2367334686a8","Type":"ContainerStarted","Data":"4b60b94263609bbc70be249dc9d5e4de7efefa42f1969a692ae52acda18d2981"}
Jul 28 20:54:06.640659 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:06.640614    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-82sks" podStartSLOduration=2.521225514 podStartE2EDuration="4.640600563s" totalImagesPullingTime="2.119375049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:54:06.638767277 +0000 UTC m=+407.256349182" watchObservedRunningTime="2026-07-28 20:54:06.640600563 +0000 UTC m=+407.258182464"
Jul 28 20:54:07.029377 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:07.029330    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:07.604678 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:07.604601    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:07.934611 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:07.934588    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:08.045185 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:08.045157    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:08.565767 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:08.565736    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:54:08.634604 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:08.634575    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9pxl8" event={"ID":"968405d0-4f21-454c-9c39-2367334686a8","Type":"ContainerStarted","Data":"1f5fedcd78e8377707e2d46db044555bf1e2050cf5b500169da8bfbf92674721"}
Jul 28 20:54:08.634758 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:08.634713    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9pxl8"
Jul 28 20:54:08.654823 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:08.654778    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9pxl8" podStartSLOduration=2.106605205 podStartE2EDuration="3.654763305s" totalImagesPullingTime="1.5481581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:54:08.653824404 +0000 UTC m=+409.271406306" watchObservedRunningTime="2026-07-28 20:54:08.654763305 +0000 UTC m=+409.272345206"
Jul 28 20:54:11.321744 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:11.321708    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:11.380416 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:11.380386    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:14.792745 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:14.792711    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:14.845965 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:14.845938    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:15.616954 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:15.616927    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:54:17.310433 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.304948    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"]
Jul 28 20:54:17.310433 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.305209    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl" podUID="e1e30820-1925-4fa0-b25a-6b52485ea72b" containerName="manager" containerID="cri-o://81c98929e1a58096bcef9fd221202af4b3f8e065de59052eb08f710db80bae38" gracePeriod=2
Jul 28 20:54:17.313043 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.313010    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"]
Jul 28 20:54:17.320727 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.320706    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"]
Jul 28 20:54:17.320952 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.320928    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj" podUID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" containerName="manager" containerID="cri-o://40db68e8a930540cdbbe6ce3320afc1efee7423c71a27b6a55cafbffff545a99" gracePeriod=2
Jul 28 20:54:17.326763 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.326728    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="e1e30820-1925-4fa0-b25a-6b52485ea72b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:54:17.329527 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.328523    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"]
Jul 28 20:54:17.329527 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.329032    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e30820-1925-4fa0-b25a-6b52485ea72b" containerName="manager"
Jul 28 20:54:17.329527 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.329050    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e30820-1925-4fa0-b25a-6b52485ea72b" containerName="manager"
Jul 28 20:54:17.329527 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.329158    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1e30820-1925-4fa0-b25a-6b52485ea72b" containerName="manager"
Jul 28 20:54:17.330230 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.330197    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cn5fj\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:54:17.332162 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.332137    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"]
Jul 28 20:54:17.332323 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.332247    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"
Jul 28 20:54:17.334838 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.334811    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="e1e30820-1925-4fa0-b25a-6b52485ea72b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:54:17.345014 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.344991    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hh2m8"]
Jul 28 20:54:17.345449 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.345428    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" containerName="manager"
Jul 28 20:54:17.345449 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.345450    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" containerName="manager"
Jul 28 20:54:17.345590 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.345578    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" containerName="manager"
Jul 28 20:54:17.348606 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.348589    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"]
Jul 28 20:54:17.348699 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.348685    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hh2m8"
Jul 28 20:54:17.358790 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.358764    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cn5fj\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:54:17.361205 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.361172    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="e1e30820-1925-4fa0-b25a-6b52485ea72b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:54:17.361309 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.361281    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hh2m8"]
Jul 28 20:54:17.388476 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.388442    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cn5fj\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:54:17.454544 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.454519    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4xp8\" (UniqueName: \"kubernetes.io/projected/83723061-26bf-44ac-bb56-ab2e487ec486-kube-api-access-q4xp8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w7g7l\" (UID: \"83723061-26bf-44ac-bb56-ab2e487ec486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"
Jul 28 20:54:17.454647 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.454624    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83723061-26bf-44ac-bb56-ab2e487ec486-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w7g7l\" (UID: \"83723061-26bf-44ac-bb56-ab2e487ec486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"
Jul 28 20:54:17.454688 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.454675    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77fjc\" (UniqueName: \"kubernetes.io/projected/99de2739-1bad-4816-8cc0-031dfab10755-kube-api-access-77fjc\") pod \"limitador-operator-controller-manager-859d7b78bd-hh2m8\" (UID: \"99de2739-1bad-4816-8cc0-031dfab10755\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hh2m8"
Jul 28 20:54:17.555140 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.555114    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4xp8\" (UniqueName: \"kubernetes.io/projected/83723061-26bf-44ac-bb56-ab2e487ec486-kube-api-access-q4xp8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w7g7l\" (UID: \"83723061-26bf-44ac-bb56-ab2e487ec486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"
Jul 28 20:54:17.555271 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.555180    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83723061-26bf-44ac-bb56-ab2e487ec486-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w7g7l\" (UID: \"83723061-26bf-44ac-bb56-ab2e487ec486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"
Jul 28 20:54:17.555271 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.555218    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77fjc\" (UniqueName: \"kubernetes.io/projected/99de2739-1bad-4816-8cc0-031dfab10755-kube-api-access-77fjc\") pod \"limitador-operator-controller-manager-859d7b78bd-hh2m8\" (UID: \"99de2739-1bad-4816-8cc0-031dfab10755\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hh2m8"
Jul 28 20:54:17.555580 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.555557    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83723061-26bf-44ac-bb56-ab2e487ec486-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w7g7l\" (UID: \"83723061-26bf-44ac-bb56-ab2e487ec486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"
Jul 28 20:54:17.563056 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.563036    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:54:17.565183 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.565162    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:54:17.565346 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.565326    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="e1e30820-1925-4fa0-b25a-6b52485ea72b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:54:17.565460 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.565443    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77fjc\" (UniqueName: \"kubernetes.io/projected/99de2739-1bad-4816-8cc0-031dfab10755-kube-api-access-77fjc\") pod \"limitador-operator-controller-manager-859d7b78bd-hh2m8\" (UID: \"99de2739-1bad-4816-8cc0-031dfab10755\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hh2m8"
Jul 28 20:54:17.565610 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.565596    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4xp8\" (UniqueName: \"kubernetes.io/projected/83723061-26bf-44ac-bb56-ab2e487ec486-kube-api-access-q4xp8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w7g7l\" (UID: \"83723061-26bf-44ac-bb56-ab2e487ec486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"
Jul 28 20:54:17.567581 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.567561    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cn5fj\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:54:17.569828 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.569809    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="e1e30820-1925-4fa0-b25a-6b52485ea72b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:54:17.571935 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.571918    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cn5fj\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:54:17.630881 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.630838    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-82sks"
Jul 28 20:54:17.633510 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.633485    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="e1e30820-1925-4fa0-b25a-6b52485ea72b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:54:17.635969 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.635948    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cn5fj\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:54:17.678507 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.678476    2537 generic.go:356] "Generic (PLEG): container finished" podID="e1e30820-1925-4fa0-b25a-6b52485ea72b" containerID="81c98929e1a58096bcef9fd221202af4b3f8e065de59052eb08f710db80bae38" exitCode=0
Jul 28 20:54:17.678665 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.678522    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:54:17.678665 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.678539    2537 scope.go:117] "RemoveContainer" containerID="81c98929e1a58096bcef9fd221202af4b3f8e065de59052eb08f710db80bae38"
Jul 28 20:54:17.680063 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.680036    2537 generic.go:356] "Generic (PLEG): container finished" podID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" containerID="40db68e8a930540cdbbe6ce3320afc1efee7423c71a27b6a55cafbffff545a99" exitCode=0
Jul 28 20:54:17.680165 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.680090    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:54:17.681157 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.681129    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="e1e30820-1925-4fa0-b25a-6b52485ea72b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:54:17.683618 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.683593    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cn5fj\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:54:17.685929 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.685899    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cn5fj\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:54:17.687464 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.687445    2537 scope.go:117] "RemoveContainer" containerID="81c98929e1a58096bcef9fd221202af4b3f8e065de59052eb08f710db80bae38"
Jul 28 20:54:17.687709 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.687689    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c98929e1a58096bcef9fd221202af4b3f8e065de59052eb08f710db80bae38\": container with ID starting with 81c98929e1a58096bcef9fd221202af4b3f8e065de59052eb08f710db80bae38 not found: ID does not exist" containerID="81c98929e1a58096bcef9fd221202af4b3f8e065de59052eb08f710db80bae38"
Jul 28 20:54:17.687782 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.687716    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c98929e1a58096bcef9fd221202af4b3f8e065de59052eb08f710db80bae38"} err="failed to get container status \"81c98929e1a58096bcef9fd221202af4b3f8e065de59052eb08f710db80bae38\": rpc error: code = NotFound desc = could not find container \"81c98929e1a58096bcef9fd221202af4b3f8e065de59052eb08f710db80bae38\": container with ID starting with 81c98929e1a58096bcef9fd221202af4b3f8e065de59052eb08f710db80bae38 not found: ID does not exist"
Jul 28 20:54:17.687782 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.687734    2537 scope.go:117] "RemoveContainer" containerID="40db68e8a930540cdbbe6ce3320afc1efee7423c71a27b6a55cafbffff545a99"
Jul 28 20:54:17.687917 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.687882    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="e1e30820-1925-4fa0-b25a-6b52485ea72b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:54:17.695100 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.695080    2537 scope.go:117] "RemoveContainer" containerID="40db68e8a930540cdbbe6ce3320afc1efee7423c71a27b6a55cafbffff545a99"
Jul 28 20:54:17.695332 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.695314    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40db68e8a930540cdbbe6ce3320afc1efee7423c71a27b6a55cafbffff545a99\": container with ID starting with 40db68e8a930540cdbbe6ce3320afc1efee7423c71a27b6a55cafbffff545a99 not found: ID does not exist" containerID="40db68e8a930540cdbbe6ce3320afc1efee7423c71a27b6a55cafbffff545a99"
Jul 28 20:54:17.695388 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.695338    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40db68e8a930540cdbbe6ce3320afc1efee7423c71a27b6a55cafbffff545a99"} err="failed to get container status \"40db68e8a930540cdbbe6ce3320afc1efee7423c71a27b6a55cafbffff545a99\": rpc error: code = NotFound desc = could not find container \"40db68e8a930540cdbbe6ce3320afc1efee7423c71a27b6a55cafbffff545a99\": container with ID starting with 40db68e8a930540cdbbe6ce3320afc1efee7423c71a27b6a55cafbffff545a99 not found: ID does not exist"
Jul 28 20:54:17.749685 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.749660    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"
Jul 28 20:54:17.756716 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.756699    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hh2m8"
Jul 28 20:54:17.756806 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.756765    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx7k7\" (UniqueName: \"kubernetes.io/projected/b1bed668-540f-425d-bab4-ef1d3f2f5a19-kube-api-access-bx7k7\") pod \"b1bed668-540f-425d-bab4-ef1d3f2f5a19\" (UID: \"b1bed668-540f-425d-bab4-ef1d3f2f5a19\") "
Jul 28 20:54:17.756888 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.756873    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcpkr\" (UniqueName: \"kubernetes.io/projected/e1e30820-1925-4fa0-b25a-6b52485ea72b-kube-api-access-jcpkr\") pod \"e1e30820-1925-4fa0-b25a-6b52485ea72b\" (UID: \"e1e30820-1925-4fa0-b25a-6b52485ea72b\") "
Jul 28 20:54:17.756940 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.756924    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1e30820-1925-4fa0-b25a-6b52485ea72b-extensions-socket-volume\") pod \"e1e30820-1925-4fa0-b25a-6b52485ea72b\" (UID: \"e1e30820-1925-4fa0-b25a-6b52485ea72b\") "
Jul 28 20:54:17.757518 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.757490    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e30820-1925-4fa0-b25a-6b52485ea72b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e1e30820-1925-4fa0-b25a-6b52485ea72b" (UID: "e1e30820-1925-4fa0-b25a-6b52485ea72b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:54:17.758799 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.758779    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bed668-540f-425d-bab4-ef1d3f2f5a19-kube-api-access-bx7k7" (OuterVolumeSpecName: "kube-api-access-bx7k7") pod "b1bed668-540f-425d-bab4-ef1d3f2f5a19" (UID: "b1bed668-540f-425d-bab4-ef1d3f2f5a19"). InnerVolumeSpecName "kube-api-access-bx7k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:54:17.758886 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.758827    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e30820-1925-4fa0-b25a-6b52485ea72b-kube-api-access-jcpkr" (OuterVolumeSpecName: "kube-api-access-jcpkr") pod "e1e30820-1925-4fa0-b25a-6b52485ea72b" (UID: "e1e30820-1925-4fa0-b25a-6b52485ea72b"). InnerVolumeSpecName "kube-api-access-jcpkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:54:17.858410 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.858373    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcpkr\" (UniqueName: \"kubernetes.io/projected/e1e30820-1925-4fa0-b25a-6b52485ea72b-kube-api-access-jcpkr\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:54:17.858410 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.858399    2537 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1e30820-1925-4fa0-b25a-6b52485ea72b-extensions-socket-volume\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:54:17.858410 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.858410    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx7k7\" (UniqueName: \"kubernetes.io/projected/b1bed668-540f-425d-bab4-ef1d3f2f5a19-kube-api-access-bx7k7\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:54:17.893375 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.893342    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"]
Jul 28 20:54:17.896280 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:54:17.896256    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83723061_26bf_44ac_bb56_ab2e487ec486.slice/crio-fc7cca2cc700fa4e27a73eea60e1a704505b3a6c19f7793661a5b47a4e45b9c9 WatchSource:0}: Error finding container fc7cca2cc700fa4e27a73eea60e1a704505b3a6c19f7793661a5b47a4e45b9c9: Status 404 returned error can't find the container with id fc7cca2cc700fa4e27a73eea60e1a704505b3a6c19f7793661a5b47a4e45b9c9
Jul 28 20:54:17.915609 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:17.915544    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hh2m8"]
Jul 28 20:54:17.918096 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:54:17.918072    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99de2739_1bad_4816_8cc0_031dfab10755.slice/crio-85052108bd8782468a61f637102d479a8a2fa7e54a5cfd428e123f321e9579d7 WatchSource:0}: Error finding container 85052108bd8782468a61f637102d479a8a2fa7e54a5cfd428e123f321e9579d7: Status 404 returned error can't find the container with id 85052108bd8782468a61f637102d479a8a2fa7e54a5cfd428e123f321e9579d7
Jul 28 20:54:17.990069 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.990042    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cn5fj\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:54:17.992548 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.992524    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="e1e30820-1925-4fa0-b25a-6b52485ea72b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:54:17.995078 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.995056    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-44gfl\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="e1e30820-1925-4fa0-b25a-6b52485ea72b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-44gfl"
Jul 28 20:54:17.997487 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:17.997468    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cn5fj\" is forbidden: User \"system:node:ip-10-0-133-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-14.ec2.internal' and this object" podUID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cn5fj"
Jul 28 20:54:18.022254 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:18.022214    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1bed668-540f-425d-bab4-ef1d3f2f5a19" path="/var/lib/kubelet/pods/b1bed668-540f-425d-bab4-ef1d3f2f5a19/volumes"
Jul 28 20:54:18.022707 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:18.022687    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1e30820-1925-4fa0-b25a-6b52485ea72b" path="/var/lib/kubelet/pods/e1e30820-1925-4fa0-b25a-6b52485ea72b/volumes"
Jul 28 20:54:18.687565 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:18.687472    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hh2m8" event={"ID":"99de2739-1bad-4816-8cc0-031dfab10755","Type":"ContainerStarted","Data":"b77b60965099c4f99ff45f2939e83c91b8553fe55ef63306f9f1fa851b18aa2f"}
Jul 28 20:54:18.687565 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:18.687532    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hh2m8" event={"ID":"99de2739-1bad-4816-8cc0-031dfab10755","Type":"ContainerStarted","Data":"85052108bd8782468a61f637102d479a8a2fa7e54a5cfd428e123f321e9579d7"}
Jul 28 20:54:18.687565 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:18.687564    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hh2m8"
Jul 28 20:54:18.689080 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:18.689056    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l" event={"ID":"83723061-26bf-44ac-bb56-ab2e487ec486","Type":"ContainerStarted","Data":"57460feffe2ce329a7b2c9040a6ce0ce6b47e81cac3f88f2a7b6ddf699d82fa2"}
Jul 28 20:54:18.689214 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:18.689109    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l" event={"ID":"83723061-26bf-44ac-bb56-ab2e487ec486","Type":"ContainerStarted","Data":"fc7cca2cc700fa4e27a73eea60e1a704505b3a6c19f7793661a5b47a4e45b9c9"}
Jul 28 20:54:18.689214 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:18.689137    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"
Jul 28 20:54:18.708240 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:18.708195    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hh2m8" podStartSLOduration=1.70817914 podStartE2EDuration="1.70817914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:54:18.706210577 +0000 UTC m=+419.323792481" watchObservedRunningTime="2026-07-28 20:54:18.70817914 +0000 UTC m=+419.325761046"
Jul 28 20:54:18.728689 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:18.728650    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l" podStartSLOduration=1.7286364010000002 podStartE2EDuration="1.728636401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:54:18.72715725 +0000 UTC m=+419.344739153" watchObservedRunningTime="2026-07-28 20:54:18.728636401 +0000 UTC m=+419.346218306"
Jul 28 20:54:19.644284 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:19.644252    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9pxl8"
Jul 28 20:54:19.940562 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:19.940492    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:20.043025 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:20.042996    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:22.358282 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:22.358244    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:22.458490 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:22.458460    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:27.385725 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:27.385686    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:27.499819 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:27.499767    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:28.938327 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:28.938294    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:29.038615 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:29.038575    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:29.248130 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:29.248041    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:29.702234 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:29.702204    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"
Jul 28 20:54:29.702418 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:29.702366    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hh2m8"
Jul 28 20:54:34.442808 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.442775    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"]
Jul 28 20:54:34.443308 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.442977    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l" podUID="83723061-26bf-44ac-bb56-ab2e487ec486" containerName="manager" containerID="cri-o://57460feffe2ce329a7b2c9040a6ce0ce6b47e81cac3f88f2a7b6ddf699d82fa2" gracePeriod=10
Jul 28 20:54:34.698828 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.698770    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"
Jul 28 20:54:34.766413 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.766379    2537 generic.go:356] "Generic (PLEG): container finished" podID="83723061-26bf-44ac-bb56-ab2e487ec486" containerID="57460feffe2ce329a7b2c9040a6ce0ce6b47e81cac3f88f2a7b6ddf699d82fa2" exitCode=0
Jul 28 20:54:34.766556 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.766429    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"
Jul 28 20:54:34.766556 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.766453    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l" event={"ID":"83723061-26bf-44ac-bb56-ab2e487ec486","Type":"ContainerDied","Data":"57460feffe2ce329a7b2c9040a6ce0ce6b47e81cac3f88f2a7b6ddf699d82fa2"}
Jul 28 20:54:34.766556 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.766499    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l" event={"ID":"83723061-26bf-44ac-bb56-ab2e487ec486","Type":"ContainerDied","Data":"fc7cca2cc700fa4e27a73eea60e1a704505b3a6c19f7793661a5b47a4e45b9c9"}
Jul 28 20:54:34.766556 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.766542    2537 scope.go:117] "RemoveContainer" containerID="57460feffe2ce329a7b2c9040a6ce0ce6b47e81cac3f88f2a7b6ddf699d82fa2"
Jul 28 20:54:34.774893 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.774878    2537 scope.go:117] "RemoveContainer" containerID="57460feffe2ce329a7b2c9040a6ce0ce6b47e81cac3f88f2a7b6ddf699d82fa2"
Jul 28 20:54:34.775067 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.775047    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4xp8\" (UniqueName: \"kubernetes.io/projected/83723061-26bf-44ac-bb56-ab2e487ec486-kube-api-access-q4xp8\") pod \"83723061-26bf-44ac-bb56-ab2e487ec486\" (UID: \"83723061-26bf-44ac-bb56-ab2e487ec486\") "
Jul 28 20:54:34.775156 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.775143    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83723061-26bf-44ac-bb56-ab2e487ec486-extensions-socket-volume\") pod \"83723061-26bf-44ac-bb56-ab2e487ec486\" (UID: \"83723061-26bf-44ac-bb56-ab2e487ec486\") "
Jul 28 20:54:34.775195 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:54:34.775160    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57460feffe2ce329a7b2c9040a6ce0ce6b47e81cac3f88f2a7b6ddf699d82fa2\": container with ID starting with 57460feffe2ce329a7b2c9040a6ce0ce6b47e81cac3f88f2a7b6ddf699d82fa2 not found: ID does not exist" containerID="57460feffe2ce329a7b2c9040a6ce0ce6b47e81cac3f88f2a7b6ddf699d82fa2"
Jul 28 20:54:34.775239 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.775191    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57460feffe2ce329a7b2c9040a6ce0ce6b47e81cac3f88f2a7b6ddf699d82fa2"} err="failed to get container status \"57460feffe2ce329a7b2c9040a6ce0ce6b47e81cac3f88f2a7b6ddf699d82fa2\": rpc error: code = NotFound desc = could not find container \"57460feffe2ce329a7b2c9040a6ce0ce6b47e81cac3f88f2a7b6ddf699d82fa2\": container with ID starting with 57460feffe2ce329a7b2c9040a6ce0ce6b47e81cac3f88f2a7b6ddf699d82fa2 not found: ID does not exist"
Jul 28 20:54:34.775453 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.775432    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83723061-26bf-44ac-bb56-ab2e487ec486-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "83723061-26bf-44ac-bb56-ab2e487ec486" (UID: "83723061-26bf-44ac-bb56-ab2e487ec486"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:54:34.776891 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.776847    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83723061-26bf-44ac-bb56-ab2e487ec486-kube-api-access-q4xp8" (OuterVolumeSpecName: "kube-api-access-q4xp8") pod "83723061-26bf-44ac-bb56-ab2e487ec486" (UID: "83723061-26bf-44ac-bb56-ab2e487ec486"). InnerVolumeSpecName "kube-api-access-q4xp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:54:34.876608 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.876575    2537 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83723061-26bf-44ac-bb56-ab2e487ec486-extensions-socket-volume\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:54:34.876608 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:34.876602    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4xp8\" (UniqueName: \"kubernetes.io/projected/83723061-26bf-44ac-bb56-ab2e487ec486-kube-api-access-q4xp8\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:54:35.089166 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:35.089136    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"]
Jul 28 20:54:35.093482 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:35.093457    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w7g7l"]
Jul 28 20:54:36.020269 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:36.020228    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83723061-26bf-44ac-bb56-ab2e487ec486" path="/var/lib/kubelet/pods/83723061-26bf-44ac-bb56-ab2e487ec486/volumes"
Jul 28 20:54:37.655498 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:37.655463    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:37.756274 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:37.756243    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:40.093834 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:40.093803    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:40.150824 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:40.150791    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:40.525435 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:40.525361    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:40.626110 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:40.626084    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:40.727370 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:40.727341    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:44.451329 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:44.451254    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:44.552023 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:44.551991    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:48.072848 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:48.072818    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:48.172556 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:48.172530    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:50.685478 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.685437    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"]
Jul 28 20:54:50.686020 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.686003    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83723061-26bf-44ac-bb56-ab2e487ec486" containerName="manager"
Jul 28 20:54:50.686020 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.686022    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="83723061-26bf-44ac-bb56-ab2e487ec486" containerName="manager"
Jul 28 20:54:50.686134 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.686112    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="83723061-26bf-44ac-bb56-ab2e487ec486" containerName="manager"
Jul 28 20:54:50.689234 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.689218    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.692089 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.692073    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 20:54:50.692155 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.692075    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-s4njj\""
Jul 28 20:54:50.699892 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.699869    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"]
Jul 28 20:54:50.707412 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.707392    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:50.796062 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.796033    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b268625-85ea-4240-a31f-15047c283421-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.796197 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.796074    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b268625-85ea-4240-a31f-15047c283421-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.796197 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.796148    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b268625-85ea-4240-a31f-15047c283421-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.796296 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.796214    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkc2n\" (UniqueName: \"kubernetes.io/projected/9b268625-85ea-4240-a31f-15047c283421-kube-api-access-mkc2n\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.796296 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.796239    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b268625-85ea-4240-a31f-15047c283421-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.796296 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.796269    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b268625-85ea-4240-a31f-15047c283421-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.796400 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.796291    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b268625-85ea-4240-a31f-15047c283421-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.796400 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.796318    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b268625-85ea-4240-a31f-15047c283421-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.796400 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.796333    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b268625-85ea-4240-a31f-15047c283421-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.896930 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.896907    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b268625-85ea-4240-a31f-15047c283421-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.897070 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.896939    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b268625-85ea-4240-a31f-15047c283421-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.897070 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.896958    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b268625-85ea-4240-a31f-15047c283421-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.897070 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.896974    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b268625-85ea-4240-a31f-15047c283421-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.897070 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.897005    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b268625-85ea-4240-a31f-15047c283421-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.897070 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.897044    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b268625-85ea-4240-a31f-15047c283421-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.897330 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.897107    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b268625-85ea-4240-a31f-15047c283421-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.897330 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.897165    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkc2n\" (UniqueName: \"kubernetes.io/projected/9b268625-85ea-4240-a31f-15047c283421-kube-api-access-mkc2n\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.897330 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.897207    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b268625-85ea-4240-a31f-15047c283421-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.897494 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.897475    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b268625-85ea-4240-a31f-15047c283421-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.897568 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.897547    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b268625-85ea-4240-a31f-15047c283421-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.897630 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.897591    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b268625-85ea-4240-a31f-15047c283421-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.897723 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.897701    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b268625-85ea-4240-a31f-15047c283421-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.897778 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.897765    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b268625-85ea-4240-a31f-15047c283421-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.899245 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.899221    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b268625-85ea-4240-a31f-15047c283421-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.899473 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.899453    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b268625-85ea-4240-a31f-15047c283421-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.905707 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.905669    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b268625-85ea-4240-a31f-15047c283421-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.906026 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.906001    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkc2n\" (UniqueName: \"kubernetes.io/projected/9b268625-85ea-4240-a31f-15047c283421-kube-api-access-mkc2n\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8qz9v\" (UID: \"9b268625-85ea-4240-a31f-15047c283421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:50.999546 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:50.999484    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:54:51.088794 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:51.088751    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:51.131102 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:51.131079    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"]
Jul 28 20:54:51.132992 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:54:51.132962    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b268625_85ea_4240_a31f_15047c283421.slice/crio-ae1614c0bbf3ee66ba12448c9df9e6c37058c2dcf64735fb17fb82d0808685f8 WatchSource:0}: Error finding container ae1614c0bbf3ee66ba12448c9df9e6c37058c2dcf64735fb17fb82d0808685f8: Status 404 returned error can't find the container with id ae1614c0bbf3ee66ba12448c9df9e6c37058c2dcf64735fb17fb82d0808685f8
Jul 28 20:54:51.839175 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:51.839143    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v" event={"ID":"9b268625-85ea-4240-a31f-15047c283421","Type":"ContainerStarted","Data":"ae1614c0bbf3ee66ba12448c9df9e6c37058c2dcf64735fb17fb82d0808685f8"}
Jul 28 20:54:57.859535 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:57.859496    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:57.960807 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:57.960750    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:58.257388 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:58.257307    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:58.320438 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:54:58.320410    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:02.055601 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:02.055566    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:02.159508 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:02.159476    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:05.738969 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:05.738922    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 20:55:05.739250 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:05.739004    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 20:55:05.739250 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:05.739032    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 20:55:05.901606 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:05.901562    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v" event={"ID":"9b268625-85ea-4240-a31f-15047c283421","Type":"ContainerStarted","Data":"a8720352b1516ef74f62ea5ce1a9c18e1b2248e75f43848789eb4199932f1a62"}
Jul 28 20:55:05.924497 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:05.924448    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v" podStartSLOduration=1.320755546 podStartE2EDuration="15.924433024s" totalImagesPullingTime="14.603677478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:05.92305507 +0000 UTC m=+466.540636984" watchObservedRunningTime="2026-07-28 20:55:05.924433024 +0000 UTC m=+466.542014925"
Jul 28 20:55:05.999747 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:05.999678    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:55:05.999747 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:05.999730    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:55:06.926594 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:06.926558    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:06.927062 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:06.927042    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"]
Jul 28 20:55:06.930442 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:06.930425    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"
Jul 28 20:55:06.932982 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:06.932962    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 20:55:06.933121 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:06.932999    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m9l2k\""
Jul 28 20:55:06.938076 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:06.938058    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"]
Jul 28 20:55:07.005201 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.005172    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:55:07.019185 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.019157    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7253111d-803b-4a0b-9510-4a0e44cd26a5-config-file\") pod \"limitador-limitador-64c94fd64f-6fq9s\" (UID: \"7253111d-803b-4a0b-9510-4a0e44cd26a5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"
Jul 28 20:55:07.019344 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.019198    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzk4\" (UniqueName: \"kubernetes.io/projected/7253111d-803b-4a0b-9510-4a0e44cd26a5-kube-api-access-nlzk4\") pod \"limitador-limitador-64c94fd64f-6fq9s\" (UID: \"7253111d-803b-4a0b-9510-4a0e44cd26a5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"
Jul 28 20:55:07.033583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.033549    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"]
Jul 28 20:55:07.064029 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.063988    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:07.120377 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.120342    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzk4\" (UniqueName: \"kubernetes.io/projected/7253111d-803b-4a0b-9510-4a0e44cd26a5-kube-api-access-nlzk4\") pod \"limitador-limitador-64c94fd64f-6fq9s\" (UID: \"7253111d-803b-4a0b-9510-4a0e44cd26a5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"
Jul 28 20:55:07.120503 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.120437    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7253111d-803b-4a0b-9510-4a0e44cd26a5-config-file\") pod \"limitador-limitador-64c94fd64f-6fq9s\" (UID: \"7253111d-803b-4a0b-9510-4a0e44cd26a5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"
Jul 28 20:55:07.121239 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.121214    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7253111d-803b-4a0b-9510-4a0e44cd26a5-config-file\") pod \"limitador-limitador-64c94fd64f-6fq9s\" (UID: \"7253111d-803b-4a0b-9510-4a0e44cd26a5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"
Jul 28 20:55:07.121346 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.121263    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:07.128665 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.128640    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzk4\" (UniqueName: \"kubernetes.io/projected/7253111d-803b-4a0b-9510-4a0e44cd26a5-kube-api-access-nlzk4\") pod \"limitador-limitador-64c94fd64f-6fq9s\" (UID: \"7253111d-803b-4a0b-9510-4a0e44cd26a5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"
Jul 28 20:55:07.241266 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.241179    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"
Jul 28 20:55:07.370746 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.370719    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"]
Jul 28 20:55:07.373183 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:55:07.373152    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7253111d_803b_4a0b_9510_4a0e44cd26a5.slice/crio-e6ea52803d500718d1c363778fda03a3c1a8ad7621699bbb4b205db5db60e7fb WatchSource:0}: Error finding container e6ea52803d500718d1c363778fda03a3c1a8ad7621699bbb4b205db5db60e7fb: Status 404 returned error can't find the container with id e6ea52803d500718d1c363778fda03a3c1a8ad7621699bbb4b205db5db60e7fb
Jul 28 20:55:07.743829 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.743793    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-l4wfn"]
Jul 28 20:55:07.748205 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.748189    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l4wfn"
Jul 28 20:55:07.751234 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.751202    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2699m\""
Jul 28 20:55:07.757262 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.757243    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-l4wfn"]
Jul 28 20:55:07.775328 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.775308    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:07.825800 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.825742    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scgfs\" (UniqueName: \"kubernetes.io/projected/7899b170-640d-4a61-9b06-abb91c0285e9-kube-api-access-scgfs\") pod \"authorino-58cd955675-l4wfn\" (UID: \"7899b170-640d-4a61-9b06-abb91c0285e9\") " pod="kuadrant-system/authorino-58cd955675-l4wfn"
Jul 28 20:55:07.875868 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.875827    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:07.918709 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.918674    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s" event={"ID":"7253111d-803b-4a0b-9510-4a0e44cd26a5","Type":"ContainerStarted","Data":"e6ea52803d500718d1c363778fda03a3c1a8ad7621699bbb4b205db5db60e7fb"}
Jul 28 20:55:07.919583 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.919567    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:55:07.927089 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.927056    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scgfs\" (UniqueName: \"kubernetes.io/projected/7899b170-640d-4a61-9b06-abb91c0285e9-kube-api-access-scgfs\") pod \"authorino-58cd955675-l4wfn\" (UID: \"7899b170-640d-4a61-9b06-abb91c0285e9\") " pod="kuadrant-system/authorino-58cd955675-l4wfn"
Jul 28 20:55:07.952678 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.949820    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scgfs\" (UniqueName: \"kubernetes.io/projected/7899b170-640d-4a61-9b06-abb91c0285e9-kube-api-access-scgfs\") pod \"authorino-58cd955675-l4wfn\" (UID: \"7899b170-640d-4a61-9b06-abb91c0285e9\") " pod="kuadrant-system/authorino-58cd955675-l4wfn"
Jul 28 20:55:07.952678 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.950650    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-w7wj2"]
Jul 28 20:55:07.972193 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.966175    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-w7wj2"
Jul 28 20:55:07.972193 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:07.971469    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-w7wj2"]
Jul 28 20:55:08.029773 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:08.029070    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv8tf\" (UniqueName: \"kubernetes.io/projected/ee4f9025-5a61-4607-9586-3fec7a2b0c29-kube-api-access-rv8tf\") pod \"authorino-75d556bd77-w7wj2\" (UID: \"ee4f9025-5a61-4607-9586-3fec7a2b0c29\") " pod="kuadrant-system/authorino-75d556bd77-w7wj2"
Jul 28 20:55:08.059721 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:08.058663    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l4wfn"
Jul 28 20:55:08.129799 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:08.129761    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv8tf\" (UniqueName: \"kubernetes.io/projected/ee4f9025-5a61-4607-9586-3fec7a2b0c29-kube-api-access-rv8tf\") pod \"authorino-75d556bd77-w7wj2\" (UID: \"ee4f9025-5a61-4607-9586-3fec7a2b0c29\") " pod="kuadrant-system/authorino-75d556bd77-w7wj2"
Jul 28 20:55:08.143040 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:08.143009    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv8tf\" (UniqueName: \"kubernetes.io/projected/ee4f9025-5a61-4607-9586-3fec7a2b0c29-kube-api-access-rv8tf\") pod \"authorino-75d556bd77-w7wj2\" (UID: \"ee4f9025-5a61-4607-9586-3fec7a2b0c29\") " pod="kuadrant-system/authorino-75d556bd77-w7wj2"
Jul 28 20:55:08.244162 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:08.244121    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-l4wfn"]
Jul 28 20:55:08.247995 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:55:08.247964    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7899b170_640d_4a61_9b06_abb91c0285e9.slice/crio-8625566f925c061b31366963711bbe6782245e0f6d10c04d1f123209c766c618 WatchSource:0}: Error finding container 8625566f925c061b31366963711bbe6782245e0f6d10c04d1f123209c766c618: Status 404 returned error can't find the container with id 8625566f925c061b31366963711bbe6782245e0f6d10c04d1f123209c766c618
Jul 28 20:55:08.290445 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:08.290392    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-w7wj2"
Jul 28 20:55:08.438177 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:08.438136    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-w7wj2"]
Jul 28 20:55:08.441189 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:55:08.441155    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee4f9025_5a61_4607_9586_3fec7a2b0c29.slice/crio-d5a21088e43c861c40e1b9ddbccc3890ff4392193041f4aaf850937bec8cf12f WatchSource:0}: Error finding container d5a21088e43c861c40e1b9ddbccc3890ff4392193041f4aaf850937bec8cf12f: Status 404 returned error can't find the container with id d5a21088e43c861c40e1b9ddbccc3890ff4392193041f4aaf850937bec8cf12f
Jul 28 20:55:08.925230 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:08.925167    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-w7wj2" event={"ID":"ee4f9025-5a61-4607-9586-3fec7a2b0c29","Type":"ContainerStarted","Data":"d5a21088e43c861c40e1b9ddbccc3890ff4392193041f4aaf850937bec8cf12f"}
Jul 28 20:55:08.926886 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:08.926837    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l4wfn" event={"ID":"7899b170-640d-4a61-9b06-abb91c0285e9","Type":"ContainerStarted","Data":"8625566f925c061b31366963711bbe6782245e0f6d10c04d1f123209c766c618"}
Jul 28 20:55:10.941117 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:10.941065    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s" event={"ID":"7253111d-803b-4a0b-9510-4a0e44cd26a5","Type":"ContainerStarted","Data":"a0fe824c92d04e47547413368867d7afd121900e81aeadbc0e4de6b81c42aaa7"}
Jul 28 20:55:10.941567 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:10.941230    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"
Jul 28 20:55:10.958822 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:10.958771    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s" podStartSLOduration=1.906277607 podStartE2EDuration="4.95875371s" totalImagesPullingTime="3.052476103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:10.957528251 +0000 UTC m=+471.575110147" watchObservedRunningTime="2026-07-28 20:55:10.95875371 +0000 UTC m=+471.576335611"
Jul 28 20:55:12.952539 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:12.952499    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-w7wj2" event={"ID":"ee4f9025-5a61-4607-9586-3fec7a2b0c29","Type":"ContainerStarted","Data":"d41787757d4b70118fd584abb0ab8e44426ed5b08f060b2aecf36c9b8a6b4255"}
Jul 28 20:55:12.953998 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:12.953973    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l4wfn" event={"ID":"7899b170-640d-4a61-9b06-abb91c0285e9","Type":"ContainerStarted","Data":"e184da9c3b6d522107ab655056f3ad86914dd4ee766a1e1447936a0af9268a8f"}
Jul 28 20:55:12.980253 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:12.980205    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-l4wfn" podStartSLOduration=1.902771124 podStartE2EDuration="5.980190045s" totalImagesPullingTime="4.077418921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:12.979956655 +0000 UTC m=+473.597538572" watchObservedRunningTime="2026-07-28 20:55:12.980190045 +0000 UTC m=+473.597771951"
Jul 28 20:55:12.981016 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:12.980980    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-w7wj2" podStartSLOduration=2.086041808 podStartE2EDuration="5.98097035s" totalImagesPullingTime="3.894928542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:12.967243935 +0000 UTC m=+473.584825837" watchObservedRunningTime="2026-07-28 20:55:12.98097035 +0000 UTC m=+473.598552252"
Jul 28 20:55:13.007822 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:13.007792    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-l4wfn"]
Jul 28 20:55:14.962235 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:14.962170    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-l4wfn" podUID="7899b170-640d-4a61-9b06-abb91c0285e9" containerName="authorino" containerID="cri-o://e184da9c3b6d522107ab655056f3ad86914dd4ee766a1e1447936a0af9268a8f" gracePeriod=30
Jul 28 20:55:15.202043 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:15.202021    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l4wfn"
Jul 28 20:55:15.294171 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:15.294089    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scgfs\" (UniqueName: \"kubernetes.io/projected/7899b170-640d-4a61-9b06-abb91c0285e9-kube-api-access-scgfs\") pod \"7899b170-640d-4a61-9b06-abb91c0285e9\" (UID: \"7899b170-640d-4a61-9b06-abb91c0285e9\") "
Jul 28 20:55:15.296083 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:15.296054    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7899b170-640d-4a61-9b06-abb91c0285e9-kube-api-access-scgfs" (OuterVolumeSpecName: "kube-api-access-scgfs") pod "7899b170-640d-4a61-9b06-abb91c0285e9" (UID: "7899b170-640d-4a61-9b06-abb91c0285e9"). InnerVolumeSpecName "kube-api-access-scgfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:55:15.395639 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:15.395613    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scgfs\" (UniqueName: \"kubernetes.io/projected/7899b170-640d-4a61-9b06-abb91c0285e9-kube-api-access-scgfs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:55:15.968079 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:15.968044    2537 generic.go:356] "Generic (PLEG): container finished" podID="7899b170-640d-4a61-9b06-abb91c0285e9" containerID="e184da9c3b6d522107ab655056f3ad86914dd4ee766a1e1447936a0af9268a8f" exitCode=0
Jul 28 20:55:15.968551 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:15.968097    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l4wfn"
Jul 28 20:55:15.968551 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:15.968130    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l4wfn" event={"ID":"7899b170-640d-4a61-9b06-abb91c0285e9","Type":"ContainerDied","Data":"e184da9c3b6d522107ab655056f3ad86914dd4ee766a1e1447936a0af9268a8f"}
Jul 28 20:55:15.968551 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:15.968181    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l4wfn" event={"ID":"7899b170-640d-4a61-9b06-abb91c0285e9","Type":"ContainerDied","Data":"8625566f925c061b31366963711bbe6782245e0f6d10c04d1f123209c766c618"}
Jul 28 20:55:15.968551 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:15.968202    2537 scope.go:117] "RemoveContainer" containerID="e184da9c3b6d522107ab655056f3ad86914dd4ee766a1e1447936a0af9268a8f"
Jul 28 20:55:15.976842 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:15.976825    2537 scope.go:117] "RemoveContainer" containerID="e184da9c3b6d522107ab655056f3ad86914dd4ee766a1e1447936a0af9268a8f"
Jul 28 20:55:15.977154 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:55:15.977132    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e184da9c3b6d522107ab655056f3ad86914dd4ee766a1e1447936a0af9268a8f\": container with ID starting with e184da9c3b6d522107ab655056f3ad86914dd4ee766a1e1447936a0af9268a8f not found: ID does not exist" containerID="e184da9c3b6d522107ab655056f3ad86914dd4ee766a1e1447936a0af9268a8f"
Jul 28 20:55:15.977254 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:15.977159    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e184da9c3b6d522107ab655056f3ad86914dd4ee766a1e1447936a0af9268a8f"} err="failed to get container status \"e184da9c3b6d522107ab655056f3ad86914dd4ee766a1e1447936a0af9268a8f\": rpc error: code = NotFound desc = could not find container \"e184da9c3b6d522107ab655056f3ad86914dd4ee766a1e1447936a0af9268a8f\": container with ID starting with e184da9c3b6d522107ab655056f3ad86914dd4ee766a1e1447936a0af9268a8f not found: ID does not exist"
Jul 28 20:55:15.989370 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:15.989351    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-l4wfn"]
Jul 28 20:55:15.993060 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:15.993041    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-l4wfn"]
Jul 28 20:55:16.020550 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:16.020527    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7899b170-640d-4a61-9b06-abb91c0285e9" path="/var/lib/kubelet/pods/7899b170-640d-4a61-9b06-abb91c0285e9/volumes"
Jul 28 20:55:21.589724 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:21.589691    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"]
Jul 28 20:55:21.590155 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:21.589889    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s" podUID="7253111d-803b-4a0b-9510-4a0e44cd26a5" containerName="limitador" containerID="cri-o://a0fe824c92d04e47547413368867d7afd121900e81aeadbc0e4de6b81c42aaa7" gracePeriod=30
Jul 28 20:55:21.590544 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:21.590490    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"
Jul 28 20:55:22.143303 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:22.143277    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"
Jul 28 20:55:22.250764 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:22.250686    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7253111d-803b-4a0b-9510-4a0e44cd26a5-config-file\") pod \"7253111d-803b-4a0b-9510-4a0e44cd26a5\" (UID: \"7253111d-803b-4a0b-9510-4a0e44cd26a5\") "
Jul 28 20:55:22.250943 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:22.250803    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlzk4\" (UniqueName: \"kubernetes.io/projected/7253111d-803b-4a0b-9510-4a0e44cd26a5-kube-api-access-nlzk4\") pod \"7253111d-803b-4a0b-9510-4a0e44cd26a5\" (UID: \"7253111d-803b-4a0b-9510-4a0e44cd26a5\") "
Jul 28 20:55:22.251036 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:22.251014    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7253111d-803b-4a0b-9510-4a0e44cd26a5-config-file" (OuterVolumeSpecName: "config-file") pod "7253111d-803b-4a0b-9510-4a0e44cd26a5" (UID: "7253111d-803b-4a0b-9510-4a0e44cd26a5"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:55:22.252752 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:22.252726    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7253111d-803b-4a0b-9510-4a0e44cd26a5-kube-api-access-nlzk4" (OuterVolumeSpecName: "kube-api-access-nlzk4") pod "7253111d-803b-4a0b-9510-4a0e44cd26a5" (UID: "7253111d-803b-4a0b-9510-4a0e44cd26a5"). InnerVolumeSpecName "kube-api-access-nlzk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:55:22.352115 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:22.352082    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlzk4\" (UniqueName: \"kubernetes.io/projected/7253111d-803b-4a0b-9510-4a0e44cd26a5-kube-api-access-nlzk4\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:55:22.352115 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:22.352108    2537 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7253111d-803b-4a0b-9510-4a0e44cd26a5-config-file\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:55:22.999205 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:22.999169    2537 generic.go:356] "Generic (PLEG): container finished" podID="7253111d-803b-4a0b-9510-4a0e44cd26a5" containerID="a0fe824c92d04e47547413368867d7afd121900e81aeadbc0e4de6b81c42aaa7" exitCode=0
Jul 28 20:55:22.999618 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:22.999219    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"
Jul 28 20:55:22.999618 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:22.999246    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s" event={"ID":"7253111d-803b-4a0b-9510-4a0e44cd26a5","Type":"ContainerDied","Data":"a0fe824c92d04e47547413368867d7afd121900e81aeadbc0e4de6b81c42aaa7"}
Jul 28 20:55:22.999618 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:22.999301    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6fq9s" event={"ID":"7253111d-803b-4a0b-9510-4a0e44cd26a5","Type":"ContainerDied","Data":"e6ea52803d500718d1c363778fda03a3c1a8ad7621699bbb4b205db5db60e7fb"}
Jul 28 20:55:22.999618 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:22.999324    2537 scope.go:117] "RemoveContainer" containerID="a0fe824c92d04e47547413368867d7afd121900e81aeadbc0e4de6b81c42aaa7"
Jul 28 20:55:23.007726 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.007708    2537 scope.go:117] "RemoveContainer" containerID="a0fe824c92d04e47547413368867d7afd121900e81aeadbc0e4de6b81c42aaa7"
Jul 28 20:55:23.008016 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:55:23.007997    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0fe824c92d04e47547413368867d7afd121900e81aeadbc0e4de6b81c42aaa7\": container with ID starting with a0fe824c92d04e47547413368867d7afd121900e81aeadbc0e4de6b81c42aaa7 not found: ID does not exist" containerID="a0fe824c92d04e47547413368867d7afd121900e81aeadbc0e4de6b81c42aaa7"
Jul 28 20:55:23.008099 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.008023    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0fe824c92d04e47547413368867d7afd121900e81aeadbc0e4de6b81c42aaa7"} err="failed to get container status \"a0fe824c92d04e47547413368867d7afd121900e81aeadbc0e4de6b81c42aaa7\": rpc error: code = NotFound desc = could not find container \"a0fe824c92d04e47547413368867d7afd121900e81aeadbc0e4de6b81c42aaa7\": container with ID starting with a0fe824c92d04e47547413368867d7afd121900e81aeadbc0e4de6b81c42aaa7 not found: ID does not exist"
Jul 28 20:55:23.029603 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.029579    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"]
Jul 28 20:55:23.037345 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.037319    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6fq9s"]
Jul 28 20:55:23.055252 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.055232    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:23.155775 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.155750    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:23.478299 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.478260    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d7578"]
Jul 28 20:55:23.478646 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.478632    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7253111d-803b-4a0b-9510-4a0e44cd26a5" containerName="limitador"
Jul 28 20:55:23.478693 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.478648    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="7253111d-803b-4a0b-9510-4a0e44cd26a5" containerName="limitador"
Jul 28 20:55:23.478693 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.478659    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7899b170-640d-4a61-9b06-abb91c0285e9" containerName="authorino"
Jul 28 20:55:23.478693 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.478664    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="7899b170-640d-4a61-9b06-abb91c0285e9" containerName="authorino"
Jul 28 20:55:23.478788 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.478745    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="7899b170-640d-4a61-9b06-abb91c0285e9" containerName="authorino"
Jul 28 20:55:23.478788 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.478759    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="7253111d-803b-4a0b-9510-4a0e44cd26a5" containerName="limitador"
Jul 28 20:55:23.482996 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.482976    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7578"
Jul 28 20:55:23.485509 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.485484    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 28 20:55:23.485613 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.485571    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:55:23.485678 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.485627    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:55:23.486757 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.486738    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vcsgg\""
Jul 28 20:55:23.488697 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.488675    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d7578"]
Jul 28 20:55:23.502155 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.502128    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:23.561342 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.561316    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z8c6\" (UniqueName: \"kubernetes.io/projected/169e9d37-e577-4211-a26b-ef367e1d06b6-kube-api-access-7z8c6\") pod \"postgres-69fb4594cf-d7578\" (UID: \"169e9d37-e577-4211-a26b-ef367e1d06b6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7578"
Jul 28 20:55:23.561428 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.561378    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/169e9d37-e577-4211-a26b-ef367e1d06b6-data\") pod \"postgres-69fb4594cf-d7578\" (UID: \"169e9d37-e577-4211-a26b-ef367e1d06b6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7578"
Jul 28 20:55:23.662116 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.662089    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z8c6\" (UniqueName: \"kubernetes.io/projected/169e9d37-e577-4211-a26b-ef367e1d06b6-kube-api-access-7z8c6\") pod \"postgres-69fb4594cf-d7578\" (UID: \"169e9d37-e577-4211-a26b-ef367e1d06b6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7578"
Jul 28 20:55:23.662224 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.662148    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/169e9d37-e577-4211-a26b-ef367e1d06b6-data\") pod \"postgres-69fb4594cf-d7578\" (UID: \"169e9d37-e577-4211-a26b-ef367e1d06b6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7578"
Jul 28 20:55:23.662455 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.662437    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/169e9d37-e577-4211-a26b-ef367e1d06b6-data\") pod \"postgres-69fb4594cf-d7578\" (UID: \"169e9d37-e577-4211-a26b-ef367e1d06b6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7578"
Jul 28 20:55:23.670981 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.670962    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z8c6\" (UniqueName: \"kubernetes.io/projected/169e9d37-e577-4211-a26b-ef367e1d06b6-kube-api-access-7z8c6\") pod \"postgres-69fb4594cf-d7578\" (UID: \"169e9d37-e577-4211-a26b-ef367e1d06b6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7578"
Jul 28 20:55:23.794875 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.794791    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7578"
Jul 28 20:55:23.920118 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:23.920085    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d7578"]
Jul 28 20:55:23.922982 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:55:23.922949    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169e9d37_e577_4211_a26b_ef367e1d06b6.slice/crio-ab8fdc5b33c11b08e8a0816a8fef6a6c37cf12d3dd53865350b038058cb995a8 WatchSource:0}: Error finding container ab8fdc5b33c11b08e8a0816a8fef6a6c37cf12d3dd53865350b038058cb995a8: Status 404 returned error can't find the container with id ab8fdc5b33c11b08e8a0816a8fef6a6c37cf12d3dd53865350b038058cb995a8
Jul 28 20:55:24.005875 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:24.005816    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7578" event={"ID":"169e9d37-e577-4211-a26b-ef367e1d06b6","Type":"ContainerStarted","Data":"ab8fdc5b33c11b08e8a0816a8fef6a6c37cf12d3dd53865350b038058cb995a8"}
Jul 28 20:55:24.020523 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:24.020491    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7253111d-803b-4a0b-9510-4a0e44cd26a5" path="/var/lib/kubelet/pods/7253111d-803b-4a0b-9510-4a0e44cd26a5/volumes"
Jul 28 20:55:29.034194 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:29.034119    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7578" event={"ID":"169e9d37-e577-4211-a26b-ef367e1d06b6","Type":"ContainerStarted","Data":"83786030c7e857524c7faef0270da7831b71564276cb5b55c94ab43a328591a2"}
Jul 28 20:55:29.034560 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:29.034251    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7578"
Jul 28 20:55:29.052022 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:29.051977    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7578" podStartSLOduration=1.236435031 podStartE2EDuration="6.051962832s" totalImagesPullingTime="4.815527801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:29.049740388 +0000 UTC m=+489.667322291" watchObservedRunningTime="2026-07-28 20:55:29.051962832 +0000 UTC m=+489.669544733"
Jul 28 20:55:35.065914 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:35.065888    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7578"
Jul 28 20:55:35.373927 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:35.373821    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:35.594477 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:35.594442    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-48btv"]
Jul 28 20:55:35.601131 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:35.601110    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-48btv"
Jul 28 20:55:35.604494 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:35.604473    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-48btv"]
Jul 28 20:55:35.759099 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:35.759021    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw4kp\" (UniqueName: \"kubernetes.io/projected/b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc-kube-api-access-tw4kp\") pod \"authorino-5d4b67c446-48btv\" (UID: \"b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc\") " pod="kuadrant-system/authorino-5d4b67c446-48btv"
Jul 28 20:55:35.841677 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:35.841642    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-48btv"]
Jul 28 20:55:35.841909 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:55:35.841889    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tw4kp], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-48btv" podUID="b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc"
Jul 28 20:55:35.860210 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:35.860181    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4kp\" (UniqueName: \"kubernetes.io/projected/b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc-kube-api-access-tw4kp\") pod \"authorino-5d4b67c446-48btv\" (UID: \"b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc\") " pod="kuadrant-system/authorino-5d4b67c446-48btv"
Jul 28 20:55:35.869044 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:35.869023    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69f756b49d-4ct8r"]
Jul 28 20:55:35.871845 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:35.871820    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw4kp\" (UniqueName: \"kubernetes.io/projected/b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc-kube-api-access-tw4kp\") pod \"authorino-5d4b67c446-48btv\" (UID: \"b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc\") " pod="kuadrant-system/authorino-5d4b67c446-48btv"
Jul 28 20:55:35.873467 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:35.873444    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f756b49d-4ct8r"
Jul 28 20:55:35.880508 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:35.880485    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69f756b49d-4ct8r"]
Jul 28 20:55:35.961075 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:35.961047    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vms7\" (UniqueName: \"kubernetes.io/projected/730fe92c-6128-4597-ac8c-fc72b690de00-kube-api-access-6vms7\") pod \"authorino-69f756b49d-4ct8r\" (UID: \"730fe92c-6128-4597-ac8c-fc72b690de00\") " pod="kuadrant-system/authorino-69f756b49d-4ct8r"
Jul 28 20:55:36.029291 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:36.029219    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:36.061542 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:36.061519    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vms7\" (UniqueName: \"kubernetes.io/projected/730fe92c-6128-4597-ac8c-fc72b690de00-kube-api-access-6vms7\") pod \"authorino-69f756b49d-4ct8r\" (UID: \"730fe92c-6128-4597-ac8c-fc72b690de00\") " pod="kuadrant-system/authorino-69f756b49d-4ct8r"
Jul 28 20:55:36.064426 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:36.064407    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-48btv"
Jul 28 20:55:36.071197 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:36.071175    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vms7\" (UniqueName: \"kubernetes.io/projected/730fe92c-6128-4597-ac8c-fc72b690de00-kube-api-access-6vms7\") pod \"authorino-69f756b49d-4ct8r\" (UID: \"730fe92c-6128-4597-ac8c-fc72b690de00\") " pod="kuadrant-system/authorino-69f756b49d-4ct8r"
Jul 28 20:55:36.071893 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:36.071876    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-48btv"
Jul 28 20:55:36.162806 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:36.162785    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw4kp\" (UniqueName: \"kubernetes.io/projected/b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc-kube-api-access-tw4kp\") pod \"b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc\" (UID: \"b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc\") "
Jul 28 20:55:36.164791 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:36.164768    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc-kube-api-access-tw4kp" (OuterVolumeSpecName: "kube-api-access-tw4kp") pod "b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc" (UID: "b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc"). InnerVolumeSpecName "kube-api-access-tw4kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:55:36.188066 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:36.188042    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f756b49d-4ct8r"
Jul 28 20:55:36.264416 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:36.264374    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw4kp\" (UniqueName: \"kubernetes.io/projected/b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc-kube-api-access-tw4kp\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:55:36.338696 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:36.338665    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69f756b49d-4ct8r"]
Jul 28 20:55:36.340587 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:55:36.340565    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod730fe92c_6128_4597_ac8c_fc72b690de00.slice/crio-1d544eb721bab70f98ee5a1c864f4f795699688ad7eb4880f6fb6c862197b3d5 WatchSource:0}: Error finding container 1d544eb721bab70f98ee5a1c864f4f795699688ad7eb4880f6fb6c862197b3d5: Status 404 returned error can't find the container with id 1d544eb721bab70f98ee5a1c864f4f795699688ad7eb4880f6fb6c862197b3d5
Jul 28 20:55:37.071550 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.071509    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69f756b49d-4ct8r" event={"ID":"730fe92c-6128-4597-ac8c-fc72b690de00","Type":"ContainerStarted","Data":"c055b89ac9143a18b045f645e4aec9ef9c4b3e6cf1d6efba10cf766e6c677a36"}
Jul 28 20:55:37.071550 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.071531    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-48btv"
Jul 28 20:55:37.071550 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.071555    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69f756b49d-4ct8r" event={"ID":"730fe92c-6128-4597-ac8c-fc72b690de00","Type":"ContainerStarted","Data":"1d544eb721bab70f98ee5a1c864f4f795699688ad7eb4880f6fb6c862197b3d5"}
Jul 28 20:55:37.091713 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.091673    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-69f756b49d-4ct8r" podStartSLOduration=1.6442325819999999 podStartE2EDuration="2.091659737s" totalImagesPullingTime="447.427155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:37.091227058 +0000 UTC m=+497.708808960" watchObservedRunningTime="2026-07-28 20:55:37.091659737 +0000 UTC m=+497.709241638"
Jul 28 20:55:37.113130 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.113105    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-48btv"]
Jul 28 20:55:37.119314 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.119284    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-w7wj2"]
Jul 28 20:55:37.119498 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.119474    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-w7wj2" podUID="ee4f9025-5a61-4607-9586-3fec7a2b0c29" containerName="authorino" containerID="cri-o://d41787757d4b70118fd584abb0ab8e44426ed5b08f060b2aecf36c9b8a6b4255" gracePeriod=30
Jul 28 20:55:37.120927 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.120904    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-48btv"]
Jul 28 20:55:37.304904 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.304840    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74df8f75bb-68848"]
Jul 28 20:55:37.312185 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.312153    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74df8f75bb-68848"
Jul 28 20:55:37.315038 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.315016    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 20:55:37.315981 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.315959    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74df8f75bb-68848"]
Jul 28 20:55:37.365540 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.365513    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-w7wj2"
Jul 28 20:55:37.474402 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.474373    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv8tf\" (UniqueName: \"kubernetes.io/projected/ee4f9025-5a61-4607-9586-3fec7a2b0c29-kube-api-access-rv8tf\") pod \"ee4f9025-5a61-4607-9586-3fec7a2b0c29\" (UID: \"ee4f9025-5a61-4607-9586-3fec7a2b0c29\") "
Jul 28 20:55:37.474567 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.474515    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c863143b-4500-4bcf-9ba0-8a493c222dd7-tls-cert\") pod \"authorino-74df8f75bb-68848\" (UID: \"c863143b-4500-4bcf-9ba0-8a493c222dd7\") " pod="kuadrant-system/authorino-74df8f75bb-68848"
Jul 28 20:55:37.474606 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.474580    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgz6r\" (UniqueName: \"kubernetes.io/projected/c863143b-4500-4bcf-9ba0-8a493c222dd7-kube-api-access-bgz6r\") pod \"authorino-74df8f75bb-68848\" (UID: \"c863143b-4500-4bcf-9ba0-8a493c222dd7\") " pod="kuadrant-system/authorino-74df8f75bb-68848"
Jul 28 20:55:37.476475 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.476443    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4f9025-5a61-4607-9586-3fec7a2b0c29-kube-api-access-rv8tf" (OuterVolumeSpecName: "kube-api-access-rv8tf") pod "ee4f9025-5a61-4607-9586-3fec7a2b0c29" (UID: "ee4f9025-5a61-4607-9586-3fec7a2b0c29"). InnerVolumeSpecName "kube-api-access-rv8tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:55:37.575409 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.575364    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c863143b-4500-4bcf-9ba0-8a493c222dd7-tls-cert\") pod \"authorino-74df8f75bb-68848\" (UID: \"c863143b-4500-4bcf-9ba0-8a493c222dd7\") " pod="kuadrant-system/authorino-74df8f75bb-68848"
Jul 28 20:55:37.575590 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.575465    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgz6r\" (UniqueName: \"kubernetes.io/projected/c863143b-4500-4bcf-9ba0-8a493c222dd7-kube-api-access-bgz6r\") pod \"authorino-74df8f75bb-68848\" (UID: \"c863143b-4500-4bcf-9ba0-8a493c222dd7\") " pod="kuadrant-system/authorino-74df8f75bb-68848"
Jul 28 20:55:37.575665 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.575604    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv8tf\" (UniqueName: \"kubernetes.io/projected/ee4f9025-5a61-4607-9586-3fec7a2b0c29-kube-api-access-rv8tf\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:55:37.579074 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.579041    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c863143b-4500-4bcf-9ba0-8a493c222dd7-tls-cert\") pod \"authorino-74df8f75bb-68848\" (UID: \"c863143b-4500-4bcf-9ba0-8a493c222dd7\") " pod="kuadrant-system/authorino-74df8f75bb-68848"
Jul 28 20:55:37.588880 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.586824    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgz6r\" (UniqueName: \"kubernetes.io/projected/c863143b-4500-4bcf-9ba0-8a493c222dd7-kube-api-access-bgz6r\") pod \"authorino-74df8f75bb-68848\" (UID: \"c863143b-4500-4bcf-9ba0-8a493c222dd7\") " pod="kuadrant-system/authorino-74df8f75bb-68848"
Jul 28 20:55:37.623562 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.623489    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74df8f75bb-68848"
Jul 28 20:55:37.781480 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:37.781284    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74df8f75bb-68848"]
Jul 28 20:55:37.783637 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:55:37.783592    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc863143b_4500_4bcf_9ba0_8a493c222dd7.slice/crio-19e6854317a006a106df08c8d8c4b65554d052e20e56368169518cc78d182380 WatchSource:0}: Error finding container 19e6854317a006a106df08c8d8c4b65554d052e20e56368169518cc78d182380: Status 404 returned error can't find the container with id 19e6854317a006a106df08c8d8c4b65554d052e20e56368169518cc78d182380
Jul 28 20:55:38.020494 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:38.020419    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc" path="/var/lib/kubelet/pods/b674a52b-bdf3-4d77-b4c9-cdd0cdcd6ccc/volumes"
Jul 28 20:55:38.078093 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:38.078062    2537 generic.go:356] "Generic (PLEG): container finished" podID="ee4f9025-5a61-4607-9586-3fec7a2b0c29" containerID="d41787757d4b70118fd584abb0ab8e44426ed5b08f060b2aecf36c9b8a6b4255" exitCode=0
Jul 28 20:55:38.078458 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:38.078104    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-w7wj2"
Jul 28 20:55:38.078458 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:38.078142    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-w7wj2" event={"ID":"ee4f9025-5a61-4607-9586-3fec7a2b0c29","Type":"ContainerDied","Data":"d41787757d4b70118fd584abb0ab8e44426ed5b08f060b2aecf36c9b8a6b4255"}
Jul 28 20:55:38.078458 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:38.078179    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-w7wj2" event={"ID":"ee4f9025-5a61-4607-9586-3fec7a2b0c29","Type":"ContainerDied","Data":"d5a21088e43c861c40e1b9ddbccc3890ff4392193041f4aaf850937bec8cf12f"}
Jul 28 20:55:38.078458 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:38.078197    2537 scope.go:117] "RemoveContainer" containerID="d41787757d4b70118fd584abb0ab8e44426ed5b08f060b2aecf36c9b8a6b4255"
Jul 28 20:55:38.079466 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:38.079444    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74df8f75bb-68848" event={"ID":"c863143b-4500-4bcf-9ba0-8a493c222dd7","Type":"ContainerStarted","Data":"19e6854317a006a106df08c8d8c4b65554d052e20e56368169518cc78d182380"}
Jul 28 20:55:38.086303 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:38.086279    2537 scope.go:117] "RemoveContainer" containerID="d41787757d4b70118fd584abb0ab8e44426ed5b08f060b2aecf36c9b8a6b4255"
Jul 28 20:55:38.086553 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:55:38.086532    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41787757d4b70118fd584abb0ab8e44426ed5b08f060b2aecf36c9b8a6b4255\": container with ID starting with d41787757d4b70118fd584abb0ab8e44426ed5b08f060b2aecf36c9b8a6b4255 not found: ID does not exist" containerID="d41787757d4b70118fd584abb0ab8e44426ed5b08f060b2aecf36c9b8a6b4255"
Jul 28 20:55:38.086606 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:38.086560    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41787757d4b70118fd584abb0ab8e44426ed5b08f060b2aecf36c9b8a6b4255"} err="failed to get container status \"d41787757d4b70118fd584abb0ab8e44426ed5b08f060b2aecf36c9b8a6b4255\": rpc error: code = NotFound desc = could not find container \"d41787757d4b70118fd584abb0ab8e44426ed5b08f060b2aecf36c9b8a6b4255\": container with ID starting with d41787757d4b70118fd584abb0ab8e44426ed5b08f060b2aecf36c9b8a6b4255 not found: ID does not exist"
Jul 28 20:55:38.100003 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:38.097436    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-w7wj2"]
Jul 28 20:55:38.102434 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:38.102413    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-w7wj2"]
Jul 28 20:55:39.085145 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:39.085104    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74df8f75bb-68848" event={"ID":"c863143b-4500-4bcf-9ba0-8a493c222dd7","Type":"ContainerStarted","Data":"d69d78dac598ce70e07b9f45720d8ed063fc42d3c730e0fb769846730739038a"}
Jul 28 20:55:39.101756 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:39.101709    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-74df8f75bb-68848" podStartSLOduration=1.419034457 podStartE2EDuration="2.101693982s" totalImagesPullingTime="682.659525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:39.101232884 +0000 UTC m=+499.718814787" watchObservedRunningTime="2026-07-28 20:55:39.101693982 +0000 UTC m=+499.719275884"
Jul 28 20:55:39.126671 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:39.126638    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69f756b49d-4ct8r"]
Jul 28 20:55:39.126849 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:39.126823    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-69f756b49d-4ct8r" podUID="730fe92c-6128-4597-ac8c-fc72b690de00" containerName="authorino" containerID="cri-o://c055b89ac9143a18b045f645e4aec9ef9c4b3e6cf1d6efba10cf766e6c677a36" gracePeriod=30
Jul 28 20:55:39.404750 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:39.404723    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f756b49d-4ct8r"
Jul 28 20:55:39.590387 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:39.590357    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vms7\" (UniqueName: \"kubernetes.io/projected/730fe92c-6128-4597-ac8c-fc72b690de00-kube-api-access-6vms7\") pod \"730fe92c-6128-4597-ac8c-fc72b690de00\" (UID: \"730fe92c-6128-4597-ac8c-fc72b690de00\") "
Jul 28 20:55:39.592219 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:39.592187    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730fe92c-6128-4597-ac8c-fc72b690de00-kube-api-access-6vms7" (OuterVolumeSpecName: "kube-api-access-6vms7") pod "730fe92c-6128-4597-ac8c-fc72b690de00" (UID: "730fe92c-6128-4597-ac8c-fc72b690de00"). InnerVolumeSpecName "kube-api-access-6vms7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:55:39.691318 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:39.691253    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vms7\" (UniqueName: \"kubernetes.io/projected/730fe92c-6128-4597-ac8c-fc72b690de00-kube-api-access-6vms7\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:55:40.020750 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:40.020655    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee4f9025-5a61-4607-9586-3fec7a2b0c29" path="/var/lib/kubelet/pods/ee4f9025-5a61-4607-9586-3fec7a2b0c29/volumes"
Jul 28 20:55:40.091345 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:40.091308    2537 generic.go:356] "Generic (PLEG): container finished" podID="730fe92c-6128-4597-ac8c-fc72b690de00" containerID="c055b89ac9143a18b045f645e4aec9ef9c4b3e6cf1d6efba10cf766e6c677a36" exitCode=0
Jul 28 20:55:40.091788 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:40.091351    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f756b49d-4ct8r"
Jul 28 20:55:40.091788 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:40.091389    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69f756b49d-4ct8r" event={"ID":"730fe92c-6128-4597-ac8c-fc72b690de00","Type":"ContainerDied","Data":"c055b89ac9143a18b045f645e4aec9ef9c4b3e6cf1d6efba10cf766e6c677a36"}
Jul 28 20:55:40.091788 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:40.091429    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69f756b49d-4ct8r" event={"ID":"730fe92c-6128-4597-ac8c-fc72b690de00","Type":"ContainerDied","Data":"1d544eb721bab70f98ee5a1c864f4f795699688ad7eb4880f6fb6c862197b3d5"}
Jul 28 20:55:40.091788 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:40.091446    2537 scope.go:117] "RemoveContainer" containerID="c055b89ac9143a18b045f645e4aec9ef9c4b3e6cf1d6efba10cf766e6c677a36"
Jul 28 20:55:40.099717 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:40.099687    2537 scope.go:117] "RemoveContainer" containerID="c055b89ac9143a18b045f645e4aec9ef9c4b3e6cf1d6efba10cf766e6c677a36"
Jul 28 20:55:40.099980 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:55:40.099952    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c055b89ac9143a18b045f645e4aec9ef9c4b3e6cf1d6efba10cf766e6c677a36\": container with ID starting with c055b89ac9143a18b045f645e4aec9ef9c4b3e6cf1d6efba10cf766e6c677a36 not found: ID does not exist" containerID="c055b89ac9143a18b045f645e4aec9ef9c4b3e6cf1d6efba10cf766e6c677a36"
Jul 28 20:55:40.100046 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:40.099991    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c055b89ac9143a18b045f645e4aec9ef9c4b3e6cf1d6efba10cf766e6c677a36"} err="failed to get container status \"c055b89ac9143a18b045f645e4aec9ef9c4b3e6cf1d6efba10cf766e6c677a36\": rpc error: code = NotFound desc = could not find container \"c055b89ac9143a18b045f645e4aec9ef9c4b3e6cf1d6efba10cf766e6c677a36\": container with ID starting with c055b89ac9143a18b045f645e4aec9ef9c4b3e6cf1d6efba10cf766e6c677a36 not found: ID does not exist"
Jul 28 20:55:40.107851 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:40.107824    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69f756b49d-4ct8r"]
Jul 28 20:55:40.113640 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:40.113620    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-69f756b49d-4ct8r"]
Jul 28 20:55:41.580164 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.580128    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:41.593880 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.593842    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58ff75c5f5-m9m42"]
Jul 28 20:55:41.594192 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.594180    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="730fe92c-6128-4597-ac8c-fc72b690de00" containerName="authorino"
Jul 28 20:55:41.594234 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.594194    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="730fe92c-6128-4597-ac8c-fc72b690de00" containerName="authorino"
Jul 28 20:55:41.594234 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.594211    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee4f9025-5a61-4607-9586-3fec7a2b0c29" containerName="authorino"
Jul 28 20:55:41.594234 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.594216    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee4f9025-5a61-4607-9586-3fec7a2b0c29" containerName="authorino"
Jul 28 20:55:41.594325 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.594262    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="730fe92c-6128-4597-ac8c-fc72b690de00" containerName="authorino"
Jul 28 20:55:41.594325 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.594271    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee4f9025-5a61-4607-9586-3fec7a2b0c29" containerName="authorino"
Jul 28 20:55:41.663172 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.663138    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58ff75c5f5-m9m42"]
Jul 28 20:55:41.663318 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.663248    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58ff75c5f5-m9m42"
Jul 28 20:55:41.665882 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.665844    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 20:55:41.666046 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.665911    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 20:55:41.666046 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.665917    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j4c6p\""
Jul 28 20:55:41.687917 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.683036    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:41.737161 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.737125    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:41.762600 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.762575    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8555f666d8-fmq2x"]
Jul 28 20:55:41.809200 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.809171    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10a983fa-8874-49eb-8559-42dc47999b57-webhook-cert\") pod \"maas-controller-58ff75c5f5-m9m42\" (UID: \"10a983fa-8874-49eb-8559-42dc47999b57\") " pod="opendatahub/maas-controller-58ff75c5f5-m9m42"
Jul 28 20:55:41.809322 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.809281    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tl4q\" (UniqueName: \"kubernetes.io/projected/10a983fa-8874-49eb-8559-42dc47999b57-kube-api-access-7tl4q\") pod \"maas-controller-58ff75c5f5-m9m42\" (UID: \"10a983fa-8874-49eb-8559-42dc47999b57\") " pod="opendatahub/maas-controller-58ff75c5f5-m9m42"
Jul 28 20:55:41.812926 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.812908    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8555f666d8-fmq2x"]
Jul 28 20:55:41.813009 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.812998    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8555f666d8-fmq2x"
Jul 28 20:55:41.910191 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.910127    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tl4q\" (UniqueName: \"kubernetes.io/projected/10a983fa-8874-49eb-8559-42dc47999b57-kube-api-access-7tl4q\") pod \"maas-controller-58ff75c5f5-m9m42\" (UID: \"10a983fa-8874-49eb-8559-42dc47999b57\") " pod="opendatahub/maas-controller-58ff75c5f5-m9m42"
Jul 28 20:55:41.910191 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.910164    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da442f23-1e33-4929-98ce-84f6c32b9aa0-webhook-cert\") pod \"maas-controller-8555f666d8-fmq2x\" (UID: \"da442f23-1e33-4929-98ce-84f6c32b9aa0\") " pod="opendatahub/maas-controller-8555f666d8-fmq2x"
Jul 28 20:55:41.910191 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.910184    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10a983fa-8874-49eb-8559-42dc47999b57-webhook-cert\") pod \"maas-controller-58ff75c5f5-m9m42\" (UID: \"10a983fa-8874-49eb-8559-42dc47999b57\") " pod="opendatahub/maas-controller-58ff75c5f5-m9m42"
Jul 28 20:55:41.910416 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.910250    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt6wd\" (UniqueName: \"kubernetes.io/projected/da442f23-1e33-4929-98ce-84f6c32b9aa0-kube-api-access-zt6wd\") pod \"maas-controller-8555f666d8-fmq2x\" (UID: \"da442f23-1e33-4929-98ce-84f6c32b9aa0\") " pod="opendatahub/maas-controller-8555f666d8-fmq2x"
Jul 28 20:55:41.910416 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:55:41.910329    2537 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 28 20:55:41.910416 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:55:41.910386    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a983fa-8874-49eb-8559-42dc47999b57-webhook-cert podName:10a983fa-8874-49eb-8559-42dc47999b57 nodeName:}" failed. No retries permitted until 2026-07-28 20:55:42.410368089 +0000 UTC m=+503.027949982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/10a983fa-8874-49eb-8559-42dc47999b57-webhook-cert") pod "maas-controller-58ff75c5f5-m9m42" (UID: "10a983fa-8874-49eb-8559-42dc47999b57") : secret "maas-controller-webhook-cert" not found
Jul 28 20:55:41.918392 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:41.918372    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tl4q\" (UniqueName: \"kubernetes.io/projected/10a983fa-8874-49eb-8559-42dc47999b57-kube-api-access-7tl4q\") pod \"maas-controller-58ff75c5f5-m9m42\" (UID: \"10a983fa-8874-49eb-8559-42dc47999b57\") " pod="opendatahub/maas-controller-58ff75c5f5-m9m42"
Jul 28 20:55:42.010684 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:42.010658    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt6wd\" (UniqueName: \"kubernetes.io/projected/da442f23-1e33-4929-98ce-84f6c32b9aa0-kube-api-access-zt6wd\") pod \"maas-controller-8555f666d8-fmq2x\" (UID: \"da442f23-1e33-4929-98ce-84f6c32b9aa0\") " pod="opendatahub/maas-controller-8555f666d8-fmq2x"
Jul 28 20:55:42.010796 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:42.010717    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da442f23-1e33-4929-98ce-84f6c32b9aa0-webhook-cert\") pod \"maas-controller-8555f666d8-fmq2x\" (UID: \"da442f23-1e33-4929-98ce-84f6c32b9aa0\") " pod="opendatahub/maas-controller-8555f666d8-fmq2x"
Jul 28 20:55:42.010844 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:55:42.010834    2537 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 28 20:55:42.010905 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:55:42.010899    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da442f23-1e33-4929-98ce-84f6c32b9aa0-webhook-cert podName:da442f23-1e33-4929-98ce-84f6c32b9aa0 nodeName:}" failed. No retries permitted until 2026-07-28 20:55:42.510881426 +0000 UTC m=+503.128463323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/da442f23-1e33-4929-98ce-84f6c32b9aa0-webhook-cert") pod "maas-controller-8555f666d8-fmq2x" (UID: "da442f23-1e33-4929-98ce-84f6c32b9aa0") : secret "maas-controller-webhook-cert" not found
Jul 28 20:55:42.019449 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:42.019430    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt6wd\" (UniqueName: \"kubernetes.io/projected/da442f23-1e33-4929-98ce-84f6c32b9aa0-kube-api-access-zt6wd\") pod \"maas-controller-8555f666d8-fmq2x\" (UID: \"da442f23-1e33-4929-98ce-84f6c32b9aa0\") " pod="opendatahub/maas-controller-8555f666d8-fmq2x"
Jul 28 20:55:42.020337 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:42.020315    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="730fe92c-6128-4597-ac8c-fc72b690de00" path="/var/lib/kubelet/pods/730fe92c-6128-4597-ac8c-fc72b690de00/volumes"
Jul 28 20:55:42.112488 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:42.112462    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:42.413662 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:42.413631    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10a983fa-8874-49eb-8559-42dc47999b57-webhook-cert\") pod \"maas-controller-58ff75c5f5-m9m42\" (UID: \"10a983fa-8874-49eb-8559-42dc47999b57\") " pod="opendatahub/maas-controller-58ff75c5f5-m9m42"
Jul 28 20:55:42.416045 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:42.416018    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10a983fa-8874-49eb-8559-42dc47999b57-webhook-cert\") pod \"maas-controller-58ff75c5f5-m9m42\" (UID: \"10a983fa-8874-49eb-8559-42dc47999b57\") " pod="opendatahub/maas-controller-58ff75c5f5-m9m42"
Jul 28 20:55:42.514678 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:42.514651    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da442f23-1e33-4929-98ce-84f6c32b9aa0-webhook-cert\") pod \"maas-controller-8555f666d8-fmq2x\" (UID: \"da442f23-1e33-4929-98ce-84f6c32b9aa0\") " pod="opendatahub/maas-controller-8555f666d8-fmq2x"
Jul 28 20:55:42.516984 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:42.516960    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da442f23-1e33-4929-98ce-84f6c32b9aa0-webhook-cert\") pod \"maas-controller-8555f666d8-fmq2x\" (UID: \"da442f23-1e33-4929-98ce-84f6c32b9aa0\") " pod="opendatahub/maas-controller-8555f666d8-fmq2x"
Jul 28 20:55:42.572824 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:42.572789    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58ff75c5f5-m9m42"
Jul 28 20:55:42.722367 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:42.722341    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8555f666d8-fmq2x"
Jul 28 20:55:42.744447 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:42.744427    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58ff75c5f5-m9m42"]
Jul 28 20:55:42.746520 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:55:42.746495    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a983fa_8874_49eb_8559_42dc47999b57.slice/crio-eb2666750fc1255674e74376616811cd373588a912a060a8e3c51e7fd512474a WatchSource:0}: Error finding container eb2666750fc1255674e74376616811cd373588a912a060a8e3c51e7fd512474a: Status 404 returned error can't find the container with id eb2666750fc1255674e74376616811cd373588a912a060a8e3c51e7fd512474a
Jul 28 20:55:42.872826 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:42.872801    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8555f666d8-fmq2x"]
Jul 28 20:55:42.874791 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:55:42.874767    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda442f23_1e33_4929_98ce_84f6c32b9aa0.slice/crio-6218d3ca6bef9ac36321f8ba2c419fd9e991934c00da66190c341aa825fb4f2f WatchSource:0}: Error finding container 6218d3ca6bef9ac36321f8ba2c419fd9e991934c00da66190c341aa825fb4f2f: Status 404 returned error can't find the container with id 6218d3ca6bef9ac36321f8ba2c419fd9e991934c00da66190c341aa825fb4f2f
Jul 28 20:55:43.105876 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:43.105826    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8555f666d8-fmq2x" event={"ID":"da442f23-1e33-4929-98ce-84f6c32b9aa0","Type":"ContainerStarted","Data":"6218d3ca6bef9ac36321f8ba2c419fd9e991934c00da66190c341aa825fb4f2f"}
Jul 28 20:55:43.106893 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:43.106868    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58ff75c5f5-m9m42" event={"ID":"10a983fa-8874-49eb-8559-42dc47999b57","Type":"ContainerStarted","Data":"eb2666750fc1255674e74376616811cd373588a912a060a8e3c51e7fd512474a"}
Jul 28 20:55:45.964529 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:45.964496    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:46.065238 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:46.065207    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:46.127669 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:46.127630    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8555f666d8-fmq2x" event={"ID":"da442f23-1e33-4929-98ce-84f6c32b9aa0","Type":"ContainerStarted","Data":"aa1f8ff5d860018c97eb69c431b3b7b8f1954a222a598998e86c4115621b889c"}
Jul 28 20:55:46.127891 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:46.127755    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8555f666d8-fmq2x"
Jul 28 20:55:46.129580 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:46.129551    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58ff75c5f5-m9m42" event={"ID":"10a983fa-8874-49eb-8559-42dc47999b57","Type":"ContainerStarted","Data":"1ab4051ce983e644e7f92cd3c35cfe2eee695dfeed5648be26d2ec11c6adc4cd"}
Jul 28 20:55:46.129698 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:46.129623    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58ff75c5f5-m9m42"
Jul 28 20:55:46.148028 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:46.147985    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8555f666d8-fmq2x" podStartSLOduration=2.560633686 podStartE2EDuration="5.147970639s" totalImagesPullingTime="2.587336953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:46.146478759 +0000 UTC m=+506.764060665" watchObservedRunningTime="2026-07-28 20:55:46.147970639 +0000 UTC m=+506.765552541"
Jul 28 20:55:46.165653 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:46.165609    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58ff75c5f5-m9m42" podStartSLOduration=2.457015877 podStartE2EDuration="5.165595131s" totalImagesPullingTime="2.708579254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:46.16438145 +0000 UTC m=+506.781963364" watchObservedRunningTime="2026-07-28 20:55:46.165595131 +0000 UTC m=+506.783177034"
Jul 28 20:55:57.136182 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:57.136156    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8555f666d8-fmq2x"
Jul 28 20:55:57.136546 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:57.136222    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58ff75c5f5-m9m42"
Jul 28 20:55:57.194751 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:57.194717    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58ff75c5f5-m9m42"]
Jul 28 20:55:57.194984 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:57.194925    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58ff75c5f5-m9m42" podUID="10a983fa-8874-49eb-8559-42dc47999b57" containerName="manager" containerID="cri-o://1ab4051ce983e644e7f92cd3c35cfe2eee695dfeed5648be26d2ec11c6adc4cd" gracePeriod=30
Jul 28 20:55:57.432473 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:57.432447    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58ff75c5f5-m9m42"
Jul 28 20:55:57.532065 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:57.532037    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10a983fa-8874-49eb-8559-42dc47999b57-webhook-cert\") pod \"10a983fa-8874-49eb-8559-42dc47999b57\" (UID: \"10a983fa-8874-49eb-8559-42dc47999b57\") "
Jul 28 20:55:57.532223 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:57.532127    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tl4q\" (UniqueName: \"kubernetes.io/projected/10a983fa-8874-49eb-8559-42dc47999b57-kube-api-access-7tl4q\") pod \"10a983fa-8874-49eb-8559-42dc47999b57\" (UID: \"10a983fa-8874-49eb-8559-42dc47999b57\") "
Jul 28 20:55:57.534191 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:57.534166    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a983fa-8874-49eb-8559-42dc47999b57-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "10a983fa-8874-49eb-8559-42dc47999b57" (UID: "10a983fa-8874-49eb-8559-42dc47999b57"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:55:57.534278 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:57.534171    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a983fa-8874-49eb-8559-42dc47999b57-kube-api-access-7tl4q" (OuterVolumeSpecName: "kube-api-access-7tl4q") pod "10a983fa-8874-49eb-8559-42dc47999b57" (UID: "10a983fa-8874-49eb-8559-42dc47999b57"). InnerVolumeSpecName "kube-api-access-7tl4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:55:57.633375 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:57.633346    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tl4q\" (UniqueName: \"kubernetes.io/projected/10a983fa-8874-49eb-8559-42dc47999b57-kube-api-access-7tl4q\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:55:57.633375 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:57.633372    2537 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10a983fa-8874-49eb-8559-42dc47999b57-webhook-cert\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:55:58.184662 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:58.184630    2537 generic.go:356] "Generic (PLEG): container finished" podID="10a983fa-8874-49eb-8559-42dc47999b57" containerID="1ab4051ce983e644e7f92cd3c35cfe2eee695dfeed5648be26d2ec11c6adc4cd" exitCode=0
Jul 28 20:55:58.185108 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:58.184670    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58ff75c5f5-m9m42"
Jul 28 20:55:58.185108 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:58.184692    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58ff75c5f5-m9m42" event={"ID":"10a983fa-8874-49eb-8559-42dc47999b57","Type":"ContainerDied","Data":"1ab4051ce983e644e7f92cd3c35cfe2eee695dfeed5648be26d2ec11c6adc4cd"}
Jul 28 20:55:58.185108 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:58.184740    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58ff75c5f5-m9m42" event={"ID":"10a983fa-8874-49eb-8559-42dc47999b57","Type":"ContainerDied","Data":"eb2666750fc1255674e74376616811cd373588a912a060a8e3c51e7fd512474a"}
Jul 28 20:55:58.185108 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:58.184763    2537 scope.go:117] "RemoveContainer" containerID="1ab4051ce983e644e7f92cd3c35cfe2eee695dfeed5648be26d2ec11c6adc4cd"
Jul 28 20:55:58.193078 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:58.193058    2537 scope.go:117] "RemoveContainer" containerID="1ab4051ce983e644e7f92cd3c35cfe2eee695dfeed5648be26d2ec11c6adc4cd"
Jul 28 20:55:58.193313 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:55:58.193295    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab4051ce983e644e7f92cd3c35cfe2eee695dfeed5648be26d2ec11c6adc4cd\": container with ID starting with 1ab4051ce983e644e7f92cd3c35cfe2eee695dfeed5648be26d2ec11c6adc4cd not found: ID does not exist" containerID="1ab4051ce983e644e7f92cd3c35cfe2eee695dfeed5648be26d2ec11c6adc4cd"
Jul 28 20:55:58.193373 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:58.193326    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab4051ce983e644e7f92cd3c35cfe2eee695dfeed5648be26d2ec11c6adc4cd"} err="failed to get container status \"1ab4051ce983e644e7f92cd3c35cfe2eee695dfeed5648be26d2ec11c6adc4cd\": rpc error: code = NotFound desc = could not find container \"1ab4051ce983e644e7f92cd3c35cfe2eee695dfeed5648be26d2ec11c6adc4cd\": container with ID starting with 1ab4051ce983e644e7f92cd3c35cfe2eee695dfeed5648be26d2ec11c6adc4cd not found: ID does not exist"
Jul 28 20:55:58.202422 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:58.202400    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58ff75c5f5-m9m42"]
Jul 28 20:55:58.208864 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:55:58.208832    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58ff75c5f5-m9m42"]
Jul 28 20:56:00.025725 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:00.025693    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10a983fa-8874-49eb-8559-42dc47999b57" path="/var/lib/kubelet/pods/10a983fa-8874-49eb-8559-42dc47999b57/volumes"
Jul 28 20:56:15.224248 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:15.224139    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:15.321349 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:15.321308    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:15.797381 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:15.797353    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:15.934020 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:15.933987    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"]
Jul 28 20:56:15.934344 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:15.934332    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a983fa-8874-49eb-8559-42dc47999b57" containerName="manager"
Jul 28 20:56:15.934392 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:15.934345    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a983fa-8874-49eb-8559-42dc47999b57" containerName="manager"
Jul 28 20:56:15.934426 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:15.934408    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a983fa-8874-49eb-8559-42dc47999b57" containerName="manager"
Jul 28 20:56:15.942599 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:15.942572    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"
Jul 28 20:56:15.946414 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:15.946390    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2thqx\""
Jul 28 20:56:15.946689 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:15.946390    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 28 20:56:15.949134 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:15.949113    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"]
Jul 28 20:56:15.965686 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:15.965666    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:15.986825 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:15.986799    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00544e1f-fb7f-413b-8cf1-549e66a1e3af-maas-api-tls\") pod \"maas-api-6cd98d97c9-9npjb\" (UID: \"00544e1f-fb7f-413b-8cf1-549e66a1e3af\") " pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"
Jul 28 20:56:15.986949 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:15.986851    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mch6\" (UniqueName: \"kubernetes.io/projected/00544e1f-fb7f-413b-8cf1-549e66a1e3af-kube-api-access-5mch6\") pod \"maas-api-6cd98d97c9-9npjb\" (UID: \"00544e1f-fb7f-413b-8cf1-549e66a1e3af\") " pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"
Jul 28 20:56:15.986949 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:15.986927    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00544e1f-fb7f-413b-8cf1-549e66a1e3af-openshift-service-ca\") pod \"maas-api-6cd98d97c9-9npjb\" (UID: \"00544e1f-fb7f-413b-8cf1-549e66a1e3af\") " pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"
Jul 28 20:56:16.088033 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.088010    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00544e1f-fb7f-413b-8cf1-549e66a1e3af-maas-api-tls\") pod \"maas-api-6cd98d97c9-9npjb\" (UID: \"00544e1f-fb7f-413b-8cf1-549e66a1e3af\") " pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"
Jul 28 20:56:16.088144 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.088052    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mch6\" (UniqueName: \"kubernetes.io/projected/00544e1f-fb7f-413b-8cf1-549e66a1e3af-kube-api-access-5mch6\") pod \"maas-api-6cd98d97c9-9npjb\" (UID: \"00544e1f-fb7f-413b-8cf1-549e66a1e3af\") " pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"
Jul 28 20:56:16.088263 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.088242    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00544e1f-fb7f-413b-8cf1-549e66a1e3af-openshift-service-ca\") pod \"maas-api-6cd98d97c9-9npjb\" (UID: \"00544e1f-fb7f-413b-8cf1-549e66a1e3af\") " pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"
Jul 28 20:56:16.088815 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.088793    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00544e1f-fb7f-413b-8cf1-549e66a1e3af-openshift-service-ca\") pod \"maas-api-6cd98d97c9-9npjb\" (UID: \"00544e1f-fb7f-413b-8cf1-549e66a1e3af\") " pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"
Jul 28 20:56:16.090440 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.090410    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00544e1f-fb7f-413b-8cf1-549e66a1e3af-maas-api-tls\") pod \"maas-api-6cd98d97c9-9npjb\" (UID: \"00544e1f-fb7f-413b-8cf1-549e66a1e3af\") " pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"
Jul 28 20:56:16.097336 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.097312    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mch6\" (UniqueName: \"kubernetes.io/projected/00544e1f-fb7f-413b-8cf1-549e66a1e3af-kube-api-access-5mch6\") pod \"maas-api-6cd98d97c9-9npjb\" (UID: \"00544e1f-fb7f-413b-8cf1-549e66a1e3af\") " pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"
Jul 28 20:56:16.254781 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.254748    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"
Jul 28 20:56:16.316193 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.316142    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:16.401314 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.401278    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"]
Jul 28 20:56:16.403343 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.403320    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:16.405097 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:56:16.405058    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00544e1f_fb7f_413b_8cf1_549e66a1e3af.slice/crio-ef4b2728c8e330552235e2688344a685234aefafb0adabe645c94506b4a99685 WatchSource:0}: Error finding container ef4b2728c8e330552235e2688344a685234aefafb0adabe645c94506b4a99685: Status 404 returned error can't find the container with id ef4b2728c8e330552235e2688344a685234aefafb0adabe645c94506b4a99685
Jul 28 20:56:16.423590 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.423568    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk"]
Jul 28 20:56:16.428847 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.428830    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk"
Jul 28 20:56:16.431245 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.431228    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 28 20:56:16.431359 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.431342    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bzcld\""
Jul 28 20:56:16.436332 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.436313    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk"]
Jul 28 20:56:16.459128 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.459103    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-zjcmp"]
Jul 28 20:56:16.462819 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.462805    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-zjcmp"
Jul 28 20:56:16.472786 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.472756    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-zjcmp"]
Jul 28 20:56:16.491792 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.491760    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ec6091-a220-464c-9f02-02f631a10e3e-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-jh9xk\" (UID: \"56ec6091-a220-464c-9f02-02f631a10e3e\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk"
Jul 28 20:56:16.491945 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.491903    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf2dd\" (UniqueName: \"kubernetes.io/projected/56ec6091-a220-464c-9f02-02f631a10e3e-kube-api-access-bf2dd\") pod \"payload-pre-processing-687dbb6cf6-jh9xk\" (UID: \"56ec6091-a220-464c-9f02-02f631a10e3e\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk"
Jul 28 20:56:16.592718 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.592684    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2dd\" (UniqueName: \"kubernetes.io/projected/56ec6091-a220-464c-9f02-02f631a10e3e-kube-api-access-bf2dd\") pod \"payload-pre-processing-687dbb6cf6-jh9xk\" (UID: \"56ec6091-a220-464c-9f02-02f631a10e3e\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk"
Jul 28 20:56:16.592917 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.592730    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32df0cb1-de8b-4183-a6c5-9e0e3c03a67e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-zjcmp\" (UID: \"32df0cb1-de8b-4183-a6c5-9e0e3c03a67e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-zjcmp"
Jul 28 20:56:16.592917 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.592765    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98hq4\" (UniqueName: \"kubernetes.io/projected/32df0cb1-de8b-4183-a6c5-9e0e3c03a67e-kube-api-access-98hq4\") pod \"payload-processing-5cc6cd444b-zjcmp\" (UID: \"32df0cb1-de8b-4183-a6c5-9e0e3c03a67e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-zjcmp"
Jul 28 20:56:16.592917 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.592808    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ec6091-a220-464c-9f02-02f631a10e3e-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-jh9xk\" (UID: \"56ec6091-a220-464c-9f02-02f631a10e3e\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk"
Jul 28 20:56:16.593586 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.593561    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ec6091-a220-464c-9f02-02f631a10e3e-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-jh9xk\" (UID: \"56ec6091-a220-464c-9f02-02f631a10e3e\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk"
Jul 28 20:56:16.600722 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.600670    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2dd\" (UniqueName: \"kubernetes.io/projected/56ec6091-a220-464c-9f02-02f631a10e3e-kube-api-access-bf2dd\") pod \"payload-pre-processing-687dbb6cf6-jh9xk\" (UID: \"56ec6091-a220-464c-9f02-02f631a10e3e\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk"
Jul 28 20:56:16.694226 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.694193    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32df0cb1-de8b-4183-a6c5-9e0e3c03a67e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-zjcmp\" (UID: \"32df0cb1-de8b-4183-a6c5-9e0e3c03a67e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-zjcmp"
Jul 28 20:56:16.694226 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.694231    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98hq4\" (UniqueName: \"kubernetes.io/projected/32df0cb1-de8b-4183-a6c5-9e0e3c03a67e-kube-api-access-98hq4\") pod \"payload-processing-5cc6cd444b-zjcmp\" (UID: \"32df0cb1-de8b-4183-a6c5-9e0e3c03a67e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-zjcmp"
Jul 28 20:56:16.694970 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.694946    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32df0cb1-de8b-4183-a6c5-9e0e3c03a67e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-zjcmp\" (UID: \"32df0cb1-de8b-4183-a6c5-9e0e3c03a67e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-zjcmp"
Jul 28 20:56:16.702532 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.702509    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98hq4\" (UniqueName: \"kubernetes.io/projected/32df0cb1-de8b-4183-a6c5-9e0e3c03a67e-kube-api-access-98hq4\") pod \"payload-processing-5cc6cd444b-zjcmp\" (UID: \"32df0cb1-de8b-4183-a6c5-9e0e3c03a67e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-zjcmp"
Jul 28 20:56:16.740770 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.740745    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk"
Jul 28 20:56:16.773460 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.773435    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-zjcmp"
Jul 28 20:56:16.913459 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:16.913436    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-zjcmp"]
Jul 28 20:56:16.915567 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:56:16.915536    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32df0cb1_de8b_4183_a6c5_9e0e3c03a67e.slice/crio-0db0214190b80e59cee8ea881726ccbf5843c9995ebb744f91dc9ac86679982d WatchSource:0}: Error finding container 0db0214190b80e59cee8ea881726ccbf5843c9995ebb744f91dc9ac86679982d: Status 404 returned error can't find the container with id 0db0214190b80e59cee8ea881726ccbf5843c9995ebb744f91dc9ac86679982d
Jul 28 20:56:17.101963 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:17.101930    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk"]
Jul 28 20:56:17.104162 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:56:17.104130    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ec6091_a220_464c_9f02_02f631a10e3e.slice/crio-a5c0f71ed1b5a9203bdf79b03bbdba344b04029795da5a45632bfd56927a7e48 WatchSource:0}: Error finding container a5c0f71ed1b5a9203bdf79b03bbdba344b04029795da5a45632bfd56927a7e48: Status 404 returned error can't find the container with id a5c0f71ed1b5a9203bdf79b03bbdba344b04029795da5a45632bfd56927a7e48
Jul 28 20:56:17.275188 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:17.275146    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-zjcmp" event={"ID":"32df0cb1-de8b-4183-a6c5-9e0e3c03a67e","Type":"ContainerStarted","Data":"0db0214190b80e59cee8ea881726ccbf5843c9995ebb744f91dc9ac86679982d"}
Jul 28 20:56:17.276495 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:17.276461    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk" event={"ID":"56ec6091-a220-464c-9f02-02f631a10e3e","Type":"ContainerStarted","Data":"a5c0f71ed1b5a9203bdf79b03bbdba344b04029795da5a45632bfd56927a7e48"}
Jul 28 20:56:17.277926 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:17.277867    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb" event={"ID":"00544e1f-fb7f-413b-8cf1-549e66a1e3af","Type":"ContainerStarted","Data":"ef4b2728c8e330552235e2688344a685234aefafb0adabe645c94506b4a99685"}
Jul 28 20:56:19.299930 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:19.299890    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb" event={"ID":"00544e1f-fb7f-413b-8cf1-549e66a1e3af","Type":"ContainerStarted","Data":"1d709c477535e509119492272d6c659f187d0234f7248f67b5e83fa359b91a6d"}
Jul 28 20:56:19.300345 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:19.300049    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"
Jul 28 20:56:19.325878 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:19.325365    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb" podStartSLOduration=2.158646433 podStartE2EDuration="4.325345382s" totalImagesPullingTime="2.166698949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:56:19.320214423 +0000 UTC m=+539.937796327" watchObservedRunningTime="2026-07-28 20:56:19.325345382 +0000 UTC m=+539.942927285"
Jul 28 20:56:20.148730 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:20.148632    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:20.204837 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:20.204810    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:20.305598 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:20.305565    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-zjcmp" event={"ID":"32df0cb1-de8b-4183-a6c5-9e0e3c03a67e","Type":"ContainerStarted","Data":"166bdd7390f49dc8e715b36750ca541ecde1d174fed9d16f268ef414a1035d41"}
Jul 28 20:56:20.306052 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:20.305678    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-zjcmp"
Jul 28 20:56:20.307237 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:20.307206    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk" event={"ID":"56ec6091-a220-464c-9f02-02f631a10e3e","Type":"ContainerStarted","Data":"6b72348da48153908abd145aa2920139104ab8d6310e0d610d37426623f5c00b"}
Jul 28 20:56:20.307386 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:20.307283    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk"
Jul 28 20:56:20.325807 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:20.325747    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-zjcmp" podStartSLOduration=1.895288972 podStartE2EDuration="4.325729742s" totalImagesPullingTime="2.43044077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:56:20.322096673 +0000 UTC m=+540.939678577" watchObservedRunningTime="2026-07-28 20:56:20.325729742 +0000 UTC m=+540.943311646"
Jul 28 20:56:20.338591 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:20.338541    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk" podStartSLOduration=2.087291311 podStartE2EDuration="4.338530739s" totalImagesPullingTime="2.251239428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:56:20.336793445 +0000 UTC m=+540.954375347" watchObservedRunningTime="2026-07-28 20:56:20.338530739 +0000 UTC m=+540.956112642"
Jul 28 20:56:23.960037 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:23.959996    2537 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8qz9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 28 20:56:23.960415 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:23.960068    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v" podUID="9b268625-85ea-4240-a31f-15047c283421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 20:56:24.330055 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:24.330025    2537 generic.go:356] "Generic (PLEG): container finished" podID="9b268625-85ea-4240-a31f-15047c283421" containerID="a8720352b1516ef74f62ea5ce1a9c18e1b2248e75f43848789eb4199932f1a62" exitCode=0
Jul 28 20:56:24.330244 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:24.330105    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v" event={"ID":"9b268625-85ea-4240-a31f-15047c283421","Type":"ContainerDied","Data":"a8720352b1516ef74f62ea5ce1a9c18e1b2248e75f43848789eb4199932f1a62"}
Jul 28 20:56:24.330480 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:24.330466    2537 scope.go:117] "RemoveContainer" containerID="a8720352b1516ef74f62ea5ce1a9c18e1b2248e75f43848789eb4199932f1a62"
Jul 28 20:56:24.331403 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:24.331375    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 20:56:24.331459 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:24.331445    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 20:56:24.331497 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:24.331483    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 20:56:25.326138 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:25.325199    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cd98d97c9-9npjb"
Jul 28 20:56:25.339322 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:25.339291    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v" event={"ID":"9b268625-85ea-4240-a31f-15047c283421","Type":"ContainerStarted","Data":"fbdd0716ee3ac9b91e8551fb70c644b3389ae738a1fcea80b021dbb7f65b9649"}
Jul 28 20:56:25.999936 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:25.999906    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:56:26.000113 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:26.000090    2537 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8qz9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 28 20:56:26.000161 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:26.000139    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v" podUID="9b268625-85ea-4240-a31f-15047c283421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 28 20:56:26.306513 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:26.306436    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:26.405664 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:26.405634    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:27.000602 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:27.000567    2537 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8qz9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 28 20:56:27.000766 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:27.000621    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v" podUID="9b268625-85ea-4240-a31f-15047c283421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 28 20:56:27.124231 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:27.124195    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:28.000728 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:28.000696    2537 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8qz9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 28 20:56:28.001115 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:28.000746    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v" podUID="9b268625-85ea-4240-a31f-15047c283421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 28 20:56:29.000036 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:28.999997    2537 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8qz9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 28 20:56:29.000212 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:29.000054    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v" podUID="9b268625-85ea-4240-a31f-15047c283421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 28 20:56:30.003454 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:30.003419    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:56:30.003850 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:30.003663    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:56:30.004507 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:30.004488    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8qz9v"
Jul 28 20:56:31.315482 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:31.315447    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-zjcmp"
Jul 28 20:56:31.315920 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:31.315763    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jh9xk"
Jul 28 20:56:32.963839 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:32.963808    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"]
Jul 28 20:56:32.968554 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:32.968538    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:32.973840 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:32.971283    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 20:56:32.973840 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:32.972481    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 28 20:56:32.973840 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:32.972723    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mv9qw\""
Jul 28 20:56:32.973840 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:32.972963    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 20:56:32.977346 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:32.977314    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"]
Jul 28 20:56:32.987840 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:32.987813    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:33.063266 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.063226    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.063266 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.063268    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.063499 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.063311    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.063499 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.063363    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.063499 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.063390    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p8gp\" (UniqueName: \"kubernetes.io/projected/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-kube-api-access-4p8gp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.063499 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.063425    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.164499 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.164462    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.164674 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.164517    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.164674 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.164599    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.164674 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.164652    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.164843 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.164677    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p8gp\" (UniqueName: \"kubernetes.io/projected/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-kube-api-access-4p8gp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.164843 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.164719    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.165038 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.165013    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.165144 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.165117    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.165256 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.165180    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.166944 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.166926    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.167091 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.167074    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.172623 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.172598    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p8gp\" (UniqueName: \"kubernetes.io/projected/7e45bef1-f7a2-47b0-b08e-fde66c0302b7-kube-api-access-4p8gp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2\" (UID: \"7e45bef1-f7a2-47b0-b08e-fde66c0302b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.284026 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.283951    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:33.430451 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:33.430416    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"]
Jul 28 20:56:33.433376 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:56:33.433337    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e45bef1_f7a2_47b0_b08e_fde66c0302b7.slice/crio-60ea4e0c70380eaf54ef869459f350e529b513db22c872891445831abbf440bf WatchSource:0}: Error finding container 60ea4e0c70380eaf54ef869459f350e529b513db22c872891445831abbf440bf: Status 404 returned error can't find the container with id 60ea4e0c70380eaf54ef869459f350e529b513db22c872891445831abbf440bf
Jul 28 20:56:34.384672 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:34.384640    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2" event={"ID":"7e45bef1-f7a2-47b0-b08e-fde66c0302b7","Type":"ContainerStarted","Data":"60ea4e0c70380eaf54ef869459f350e529b513db22c872891445831abbf440bf"}
Jul 28 20:56:35.391683 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:35.391602    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2" event={"ID":"7e45bef1-f7a2-47b0-b08e-fde66c0302b7","Type":"ContainerStarted","Data":"5d681761e72ac4ae91e456db28ba7593c4cbd64c73843739416897ca166f96de"}
Jul 28 20:56:35.414999 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:35.414952    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2" podStartSLOduration=1.70730964 podStartE2EDuration="3.414932159s" totalImagesPullingTime="1.707622519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:56:35.412914896 +0000 UTC m=+556.030496799" watchObservedRunningTime="2026-07-28 20:56:35.414932159 +0000 UTC m=+556.032514061"
Jul 28 20:56:38.888174 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:38.888137    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:43.284813 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:43.284778    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:43.285231 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:43.284875    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:43.297545 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:43.297523    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:43.449437 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:43.449409    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2"
Jul 28 20:56:45.962175 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:45.962139    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"]
Jul 28 20:56:45.965513 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:45.965490    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:45.968033 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:45.968012    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 28 20:56:45.977707 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:45.977678    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"]
Jul 28 20:56:45.992440 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:45.992416    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:46.065742 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.065715    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.065893 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.065846    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.065950 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.065916    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.066010 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.065995    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.066050 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.066032    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.066085 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.066055    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l96p\" (UniqueName: \"kubernetes.io/projected/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-kube-api-access-5l96p\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.166912 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.166850    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.166912 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.166911    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.167139 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.166949    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.167139 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.166971    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.167139 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.166986    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l96p\" (UniqueName: \"kubernetes.io/projected/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-kube-api-access-5l96p\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.167139 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.167045    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.167392 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.167301    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.167392 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.167336    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.167392 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.167362    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.169259 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.169231    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.169370 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.169356    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.174768 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.174744    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l96p\" (UniqueName: \"kubernetes.io/projected/5e54d3b1-49cc-4160-ab1a-50a13ff59e9b-kube-api-access-5l96p\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zw679\" (UID: \"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.279282 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.279197    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:46.618022 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:46.617998    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"]
Jul 28 20:56:46.620513 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:56:46.620488    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e54d3b1_49cc_4160_ab1a_50a13ff59e9b.slice/crio-b1f3dfbdc417aedfab1f14ea22a91a0da9d6871df5cd1a8d19d64fed9088e4d1 WatchSource:0}: Error finding container b1f3dfbdc417aedfab1f14ea22a91a0da9d6871df5cd1a8d19d64fed9088e4d1: Status 404 returned error can't find the container with id b1f3dfbdc417aedfab1f14ea22a91a0da9d6871df5cd1a8d19d64fed9088e4d1
Jul 28 20:56:47.457969 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:47.457934    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679" event={"ID":"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b","Type":"ContainerStarted","Data":"863d76f8f48c28f63835861bbb485e47d717dd41edda81ab0ef78e3c21382811"}
Jul 28 20:56:47.458349 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:47.457996    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679" event={"ID":"5e54d3b1-49cc-4160-ab1a-50a13ff59e9b","Type":"ContainerStarted","Data":"b1f3dfbdc417aedfab1f14ea22a91a0da9d6871df5cd1a8d19d64fed9088e4d1"}
Jul 28 20:56:47.481346 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:47.481297    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679" podStartSLOduration=2.292490527 podStartE2EDuration="2.481281862s" totalImagesPullingTime="188.791335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:56:47.4787438 +0000 UTC m=+568.096325702" watchObservedRunningTime="2026-07-28 20:56:47.481281862 +0000 UTC m=+568.098863764"
Jul 28 20:56:56.279328 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:56.279293    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:56.279692 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:56.279357    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:56.291614 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:56.291590    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:56:56.509206 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:56:56.509171    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zw679"
Jul 28 20:57:02.992242 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:02.992207    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:11.534714 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:11.534683    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:18.497663 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:18.497629    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:18.548098 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:18.548070    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:19.830656 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:19.830625    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 20:57:19.839658 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:19.839637    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 20:57:21.830197 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:21.830168    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:30.474588 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.474557    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"]
Jul 28 20:57:30.479336 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.479312    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.481883 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.481837    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 28 20:57:30.486769 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.486750    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"]
Jul 28 20:57:30.498015 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.497993    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:30.638898 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.638849    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16133060-8272-4e80-8877-f640a1b55eb4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.639076 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.638974    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16133060-8272-4e80-8877-f640a1b55eb4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.639076 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.639004    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wt55\" (UniqueName: \"kubernetes.io/projected/16133060-8272-4e80-8877-f640a1b55eb4-kube-api-access-4wt55\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.639193 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.639101    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16133060-8272-4e80-8877-f640a1b55eb4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.639252 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.639209    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16133060-8272-4e80-8877-f640a1b55eb4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.639252 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.639244    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16133060-8272-4e80-8877-f640a1b55eb4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.740665 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.740582    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16133060-8272-4e80-8877-f640a1b55eb4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.740665 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.740631    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16133060-8272-4e80-8877-f640a1b55eb4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.740665 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.740647    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16133060-8272-4e80-8877-f640a1b55eb4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.740665 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.740666    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16133060-8272-4e80-8877-f640a1b55eb4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.741035 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.740712    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16133060-8272-4e80-8877-f640a1b55eb4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.741035 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.740733    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wt55\" (UniqueName: \"kubernetes.io/projected/16133060-8272-4e80-8877-f640a1b55eb4-kube-api-access-4wt55\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.741155 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.741047    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16133060-8272-4e80-8877-f640a1b55eb4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.741155 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.741115    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16133060-8272-4e80-8877-f640a1b55eb4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.741258 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.741149    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16133060-8272-4e80-8877-f640a1b55eb4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.743130 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.743095    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16133060-8272-4e80-8877-f640a1b55eb4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.743538 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.743519    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16133060-8272-4e80-8877-f640a1b55eb4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.750509 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.750484    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wt55\" (UniqueName: \"kubernetes.io/projected/16133060-8272-4e80-8877-f640a1b55eb4-kube-api-access-4wt55\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q\" (UID: \"16133060-8272-4e80-8877-f640a1b55eb4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.788693 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.788669    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:30.917183 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.917158    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"]
Jul 28 20:57:30.919218 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:57:30.919189    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16133060_8272_4e80_8877_f640a1b55eb4.slice/crio-d3ed61bdbc59c4da8464e8d53153a1d5c4a6b0937007ef980ec2c0397411319b WatchSource:0}: Error finding container d3ed61bdbc59c4da8464e8d53153a1d5c4a6b0937007ef980ec2c0397411319b: Status 404 returned error can't find the container with id d3ed61bdbc59c4da8464e8d53153a1d5c4a6b0937007ef980ec2c0397411319b
Jul 28 20:57:30.920438 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:30.920421    2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:57:31.672554 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:31.672508    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q" event={"ID":"16133060-8272-4e80-8877-f640a1b55eb4","Type":"ContainerStarted","Data":"c0511730d43e3599cdd20e3d06312dcf3d76e7e1d184c081de6ea2e5ba827e47"}
Jul 28 20:57:31.672554 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:31.672553    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q" event={"ID":"16133060-8272-4e80-8877-f640a1b55eb4","Type":"ContainerStarted","Data":"d3ed61bdbc59c4da8464e8d53153a1d5c4a6b0937007ef980ec2c0397411319b"}
Jul 28 20:57:31.691108 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:31.691062    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q" podStartSLOduration=1.484594091 podStartE2EDuration="1.691046374s" totalImagesPullingTime="206.452283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:57:31.690045157 +0000 UTC m=+612.307627060" watchObservedRunningTime="2026-07-28 20:57:31.691046374 +0000 UTC m=+612.308628276"
Jul 28 20:57:40.789376 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:40.789338    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:40.789834 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:40.789393    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:40.801954 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:40.801929    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:57:41.730354 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:57:41.730317    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q"
Jul 28 20:58:15.482599 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:15.482561    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55cb49b747-rc952"]
Jul 28 20:58:15.486088 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:15.486069    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55cb49b747-rc952"
Jul 28 20:58:15.492891 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:15.492867    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55cb49b747-rc952"]
Jul 28 20:58:15.618018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:15.617983    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frcz2\" (UniqueName: \"kubernetes.io/projected/cc8e82b2-950c-447a-8749-d5c097096f05-kube-api-access-frcz2\") pod \"authorino-55cb49b747-rc952\" (UID: \"cc8e82b2-950c-447a-8749-d5c097096f05\") " pod="kuadrant-system/authorino-55cb49b747-rc952"
Jul 28 20:58:15.618183 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:15.618032    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc8e82b2-950c-447a-8749-d5c097096f05-tls-cert\") pod \"authorino-55cb49b747-rc952\" (UID: \"cc8e82b2-950c-447a-8749-d5c097096f05\") " pod="kuadrant-system/authorino-55cb49b747-rc952"
Jul 28 20:58:15.719432 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:15.719398    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frcz2\" (UniqueName: \"kubernetes.io/projected/cc8e82b2-950c-447a-8749-d5c097096f05-kube-api-access-frcz2\") pod \"authorino-55cb49b747-rc952\" (UID: \"cc8e82b2-950c-447a-8749-d5c097096f05\") " pod="kuadrant-system/authorino-55cb49b747-rc952"
Jul 28 20:58:15.719589 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:15.719447    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc8e82b2-950c-447a-8749-d5c097096f05-tls-cert\") pod \"authorino-55cb49b747-rc952\" (UID: \"cc8e82b2-950c-447a-8749-d5c097096f05\") " pod="kuadrant-system/authorino-55cb49b747-rc952"
Jul 28 20:58:15.721815 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:15.721793    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc8e82b2-950c-447a-8749-d5c097096f05-tls-cert\") pod \"authorino-55cb49b747-rc952\" (UID: \"cc8e82b2-950c-447a-8749-d5c097096f05\") " pod="kuadrant-system/authorino-55cb49b747-rc952"
Jul 28 20:58:15.728053 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:15.728031    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frcz2\" (UniqueName: \"kubernetes.io/projected/cc8e82b2-950c-447a-8749-d5c097096f05-kube-api-access-frcz2\") pod \"authorino-55cb49b747-rc952\" (UID: \"cc8e82b2-950c-447a-8749-d5c097096f05\") " pod="kuadrant-system/authorino-55cb49b747-rc952"
Jul 28 20:58:15.796220 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:15.796133    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55cb49b747-rc952"
Jul 28 20:58:15.931274 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:15.931248    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55cb49b747-rc952"]
Jul 28 20:58:15.933413 ip-10-0-133-14 kubenswrapper[2537]: W0728 20:58:15.933387    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc8e82b2_950c_447a_8749_d5c097096f05.slice/crio-fb3c8e13f65893de11301d8ceec2421f9cc76904817a969b7dc764487c8c860f WatchSource:0}: Error finding container fb3c8e13f65893de11301d8ceec2421f9cc76904817a969b7dc764487c8c860f: Status 404 returned error can't find the container with id fb3c8e13f65893de11301d8ceec2421f9cc76904817a969b7dc764487c8c860f
Jul 28 20:58:16.889231 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:16.889184    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55cb49b747-rc952" event={"ID":"cc8e82b2-950c-447a-8749-d5c097096f05","Type":"ContainerStarted","Data":"d1ddbd083b521a33a6d2ae682c734ebad16bdacb3f11b594c8075500eedbe2fc"}
Jul 28 20:58:16.889705 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:16.889244    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55cb49b747-rc952" event={"ID":"cc8e82b2-950c-447a-8749-d5c097096f05","Type":"ContainerStarted","Data":"fb3c8e13f65893de11301d8ceec2421f9cc76904817a969b7dc764487c8c860f"}
Jul 28 20:58:16.906829 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:16.906770    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55cb49b747-rc952" podStartSLOduration=1.545605637 podStartE2EDuration="1.906749641s" totalImagesPullingTime="361.144004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:58:16.906355324 +0000 UTC m=+657.523937228" watchObservedRunningTime="2026-07-28 20:58:16.906749641 +0000 UTC m=+657.524331546"
Jul 28 20:58:16.937087 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:16.937052    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74df8f75bb-68848"]
Jul 28 20:58:16.937317 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:16.937288    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-74df8f75bb-68848" podUID="c863143b-4500-4bcf-9ba0-8a493c222dd7" containerName="authorino" containerID="cri-o://d69d78dac598ce70e07b9f45720d8ed063fc42d3c730e0fb769846730739038a" gracePeriod=30
Jul 28 20:58:17.194671 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.194646    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74df8f75bb-68848"
Jul 28 20:58:17.332930 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.332900    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c863143b-4500-4bcf-9ba0-8a493c222dd7-tls-cert\") pod \"c863143b-4500-4bcf-9ba0-8a493c222dd7\" (UID: \"c863143b-4500-4bcf-9ba0-8a493c222dd7\") "
Jul 28 20:58:17.333126 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.332978    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgz6r\" (UniqueName: \"kubernetes.io/projected/c863143b-4500-4bcf-9ba0-8a493c222dd7-kube-api-access-bgz6r\") pod \"c863143b-4500-4bcf-9ba0-8a493c222dd7\" (UID: \"c863143b-4500-4bcf-9ba0-8a493c222dd7\") "
Jul 28 20:58:17.335023 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.334996    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c863143b-4500-4bcf-9ba0-8a493c222dd7-kube-api-access-bgz6r" (OuterVolumeSpecName: "kube-api-access-bgz6r") pod "c863143b-4500-4bcf-9ba0-8a493c222dd7" (UID: "c863143b-4500-4bcf-9ba0-8a493c222dd7"). InnerVolumeSpecName "kube-api-access-bgz6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:58:17.343629 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.343598    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c863143b-4500-4bcf-9ba0-8a493c222dd7-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c863143b-4500-4bcf-9ba0-8a493c222dd7" (UID: "c863143b-4500-4bcf-9ba0-8a493c222dd7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:58:17.434647 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.434563    2537 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c863143b-4500-4bcf-9ba0-8a493c222dd7-tls-cert\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:58:17.434647 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.434595    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgz6r\" (UniqueName: \"kubernetes.io/projected/c863143b-4500-4bcf-9ba0-8a493c222dd7-kube-api-access-bgz6r\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:58:17.458147 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.458114    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:58:17.566912 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.566848    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:58:17.899564 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.899523    2537 generic.go:356] "Generic (PLEG): container finished" podID="c863143b-4500-4bcf-9ba0-8a493c222dd7" containerID="d69d78dac598ce70e07b9f45720d8ed063fc42d3c730e0fb769846730739038a" exitCode=0
Jul 28 20:58:17.900059 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.899636    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74df8f75bb-68848"
Jul 28 20:58:17.900059 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.899679    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74df8f75bb-68848" event={"ID":"c863143b-4500-4bcf-9ba0-8a493c222dd7","Type":"ContainerDied","Data":"d69d78dac598ce70e07b9f45720d8ed063fc42d3c730e0fb769846730739038a"}
Jul 28 20:58:17.900059 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.899730    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74df8f75bb-68848" event={"ID":"c863143b-4500-4bcf-9ba0-8a493c222dd7","Type":"ContainerDied","Data":"19e6854317a006a106df08c8d8c4b65554d052e20e56368169518cc78d182380"}
Jul 28 20:58:17.900059 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.899754    2537 scope.go:117] "RemoveContainer" containerID="d69d78dac598ce70e07b9f45720d8ed063fc42d3c730e0fb769846730739038a"
Jul 28 20:58:17.910426 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.910239    2537 scope.go:117] "RemoveContainer" containerID="d69d78dac598ce70e07b9f45720d8ed063fc42d3c730e0fb769846730739038a"
Jul 28 20:58:17.910561 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:58:17.910532    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69d78dac598ce70e07b9f45720d8ed063fc42d3c730e0fb769846730739038a\": container with ID starting with d69d78dac598ce70e07b9f45720d8ed063fc42d3c730e0fb769846730739038a not found: ID does not exist" containerID="d69d78dac598ce70e07b9f45720d8ed063fc42d3c730e0fb769846730739038a"
Jul 28 20:58:17.910660 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.910574    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69d78dac598ce70e07b9f45720d8ed063fc42d3c730e0fb769846730739038a"} err="failed to get container status \"d69d78dac598ce70e07b9f45720d8ed063fc42d3c730e0fb769846730739038a\": rpc error: code = NotFound desc = could not find container \"d69d78dac598ce70e07b9f45720d8ed063fc42d3c730e0fb769846730739038a\": container with ID starting with d69d78dac598ce70e07b9f45720d8ed063fc42d3c730e0fb769846730739038a not found: ID does not exist"
Jul 28 20:58:17.925067 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.925034    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74df8f75bb-68848"]
Jul 28 20:58:17.933026 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:17.932997    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-74df8f75bb-68848"]
Jul 28 20:58:18.021985 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:18.021905    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c863143b-4500-4bcf-9ba0-8a493c222dd7" path="/var/lib/kubelet/pods/c863143b-4500-4bcf-9ba0-8a493c222dd7/volumes"
Jul 28 20:58:18.562485 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:18.562448    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:58:18.664915 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:18.664883    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:58:18.873479 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:18.873399    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:58:18.973018 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:18.972985    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:58:26.339194 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:26.339152    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:58:33.301833 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:33.301789    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:58:38.450156 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:58:38.450120    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 20:59:05.312274 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:05.312242    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8555f666d8-fmq2x"]
Jul 28 20:59:05.312921 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:05.312484    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8555f666d8-fmq2x" podUID="da442f23-1e33-4929-98ce-84f6c32b9aa0" containerName="manager" containerID="cri-o://aa1f8ff5d860018c97eb69c431b3b7b8f1954a222a598998e86c4115621b889c" gracePeriod=30
Jul 28 20:59:05.567925 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:05.567869    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8555f666d8-fmq2x"
Jul 28 20:59:05.648740 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:05.648710    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da442f23-1e33-4929-98ce-84f6c32b9aa0-webhook-cert\") pod \"da442f23-1e33-4929-98ce-84f6c32b9aa0\" (UID: \"da442f23-1e33-4929-98ce-84f6c32b9aa0\") "
Jul 28 20:59:05.648915 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:05.648746    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zt6wd\" (UniqueName: \"kubernetes.io/projected/da442f23-1e33-4929-98ce-84f6c32b9aa0-kube-api-access-zt6wd\") pod \"da442f23-1e33-4929-98ce-84f6c32b9aa0\" (UID: \"da442f23-1e33-4929-98ce-84f6c32b9aa0\") "
Jul 28 20:59:05.650676 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:05.650646    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da442f23-1e33-4929-98ce-84f6c32b9aa0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "da442f23-1e33-4929-98ce-84f6c32b9aa0" (UID: "da442f23-1e33-4929-98ce-84f6c32b9aa0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:59:05.650786 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:05.650701    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da442f23-1e33-4929-98ce-84f6c32b9aa0-kube-api-access-zt6wd" (OuterVolumeSpecName: "kube-api-access-zt6wd") pod "da442f23-1e33-4929-98ce-84f6c32b9aa0" (UID: "da442f23-1e33-4929-98ce-84f6c32b9aa0"). InnerVolumeSpecName "kube-api-access-zt6wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:59:05.750016 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:05.749986    2537 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da442f23-1e33-4929-98ce-84f6c32b9aa0-webhook-cert\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:59:05.750016 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:05.750012    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt6wd\" (UniqueName: \"kubernetes.io/projected/da442f23-1e33-4929-98ce-84f6c32b9aa0-kube-api-access-zt6wd\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 20:59:06.137585 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:06.137547    2537 generic.go:356] "Generic (PLEG): container finished" podID="da442f23-1e33-4929-98ce-84f6c32b9aa0" containerID="aa1f8ff5d860018c97eb69c431b3b7b8f1954a222a598998e86c4115621b889c" exitCode=0
Jul 28 20:59:06.137764 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:06.137600    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8555f666d8-fmq2x"
Jul 28 20:59:06.137764 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:06.137600    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8555f666d8-fmq2x" event={"ID":"da442f23-1e33-4929-98ce-84f6c32b9aa0","Type":"ContainerDied","Data":"aa1f8ff5d860018c97eb69c431b3b7b8f1954a222a598998e86c4115621b889c"}
Jul 28 20:59:06.137764 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:06.137666    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8555f666d8-fmq2x" event={"ID":"da442f23-1e33-4929-98ce-84f6c32b9aa0","Type":"ContainerDied","Data":"6218d3ca6bef9ac36321f8ba2c419fd9e991934c00da66190c341aa825fb4f2f"}
Jul 28 20:59:06.137764 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:06.137699    2537 scope.go:117] "RemoveContainer" containerID="aa1f8ff5d860018c97eb69c431b3b7b8f1954a222a598998e86c4115621b889c"
Jul 28 20:59:06.146339 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:06.146324    2537 scope.go:117] "RemoveContainer" containerID="aa1f8ff5d860018c97eb69c431b3b7b8f1954a222a598998e86c4115621b889c"
Jul 28 20:59:06.146606 ip-10-0-133-14 kubenswrapper[2537]: E0728 20:59:06.146588    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1f8ff5d860018c97eb69c431b3b7b8f1954a222a598998e86c4115621b889c\": container with ID starting with aa1f8ff5d860018c97eb69c431b3b7b8f1954a222a598998e86c4115621b889c not found: ID does not exist" containerID="aa1f8ff5d860018c97eb69c431b3b7b8f1954a222a598998e86c4115621b889c"
Jul 28 20:59:06.146657 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:06.146619    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1f8ff5d860018c97eb69c431b3b7b8f1954a222a598998e86c4115621b889c"} err="failed to get container status \"aa1f8ff5d860018c97eb69c431b3b7b8f1954a222a598998e86c4115621b889c\": rpc error: code = NotFound desc = could not find container \"aa1f8ff5d860018c97eb69c431b3b7b8f1954a222a598998e86c4115621b889c\": container with ID starting with aa1f8ff5d860018c97eb69c431b3b7b8f1954a222a598998e86c4115621b889c not found: ID does not exist"
Jul 28 20:59:06.155233 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:06.155212    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8555f666d8-fmq2x"]
Jul 28 20:59:06.157074 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:06.157045    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8555f666d8-fmq2x"]
Jul 28 20:59:08.021601 ip-10-0-133-14 kubenswrapper[2537]: I0728 20:59:08.021569    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da442f23-1e33-4929-98ce-84f6c32b9aa0" path="/var/lib/kubelet/pods/da442f23-1e33-4929-98ce-84f6c32b9aa0/volumes"
Jul 28 21:00:06.933323 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:06.933284    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8555f666d8-wmv8t"]
Jul 28 21:00:06.933767 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:06.933683    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da442f23-1e33-4929-98ce-84f6c32b9aa0" containerName="manager"
Jul 28 21:00:06.933767 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:06.933697    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="da442f23-1e33-4929-98ce-84f6c32b9aa0" containerName="manager"
Jul 28 21:00:06.933767 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:06.933719    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c863143b-4500-4bcf-9ba0-8a493c222dd7" containerName="authorino"
Jul 28 21:00:06.933767 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:06.933725    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="c863143b-4500-4bcf-9ba0-8a493c222dd7" containerName="authorino"
Jul 28 21:00:06.933957 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:06.933792    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="da442f23-1e33-4929-98ce-84f6c32b9aa0" containerName="manager"
Jul 28 21:00:06.933957 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:06.933806    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="c863143b-4500-4bcf-9ba0-8a493c222dd7" containerName="authorino"
Jul 28 21:00:06.936973 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:06.936943    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8555f666d8-wmv8t"
Jul 28 21:00:06.939521 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:06.939499    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 21:00:06.939641 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:06.939545    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 21:00:06.940808 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:06.940791    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j4c6p\""
Jul 28 21:00:06.944670 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:06.944647    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8555f666d8-wmv8t"]
Jul 28 21:00:07.064097 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:07.064062    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/336dff12-2e62-4970-89b7-6e58679fe739-webhook-cert\") pod \"maas-controller-8555f666d8-wmv8t\" (UID: \"336dff12-2e62-4970-89b7-6e58679fe739\") " pod="opendatahub/maas-controller-8555f666d8-wmv8t"
Jul 28 21:00:07.064259 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:07.064107    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsf2d\" (UniqueName: \"kubernetes.io/projected/336dff12-2e62-4970-89b7-6e58679fe739-kube-api-access-zsf2d\") pod \"maas-controller-8555f666d8-wmv8t\" (UID: \"336dff12-2e62-4970-89b7-6e58679fe739\") " pod="opendatahub/maas-controller-8555f666d8-wmv8t"
Jul 28 21:00:07.165043 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:07.165012    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/336dff12-2e62-4970-89b7-6e58679fe739-webhook-cert\") pod \"maas-controller-8555f666d8-wmv8t\" (UID: \"336dff12-2e62-4970-89b7-6e58679fe739\") " pod="opendatahub/maas-controller-8555f666d8-wmv8t"
Jul 28 21:00:07.165204 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:07.165066    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsf2d\" (UniqueName: \"kubernetes.io/projected/336dff12-2e62-4970-89b7-6e58679fe739-kube-api-access-zsf2d\") pod \"maas-controller-8555f666d8-wmv8t\" (UID: \"336dff12-2e62-4970-89b7-6e58679fe739\") " pod="opendatahub/maas-controller-8555f666d8-wmv8t"
Jul 28 21:00:07.167206 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:07.167188    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/336dff12-2e62-4970-89b7-6e58679fe739-webhook-cert\") pod \"maas-controller-8555f666d8-wmv8t\" (UID: \"336dff12-2e62-4970-89b7-6e58679fe739\") " pod="opendatahub/maas-controller-8555f666d8-wmv8t"
Jul 28 21:00:07.174157 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:07.174121    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsf2d\" (UniqueName: \"kubernetes.io/projected/336dff12-2e62-4970-89b7-6e58679fe739-kube-api-access-zsf2d\") pod \"maas-controller-8555f666d8-wmv8t\" (UID: \"336dff12-2e62-4970-89b7-6e58679fe739\") " pod="opendatahub/maas-controller-8555f666d8-wmv8t"
Jul 28 21:00:07.248312 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:07.248240    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8555f666d8-wmv8t"
Jul 28 21:00:07.379607 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:07.379575    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8555f666d8-wmv8t"]
Jul 28 21:00:07.381463 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:00:07.381435    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336dff12_2e62_4970_89b7_6e58679fe739.slice/crio-bdc034ac178b7527774e421ed27c73b7ab52978e9f419f062bfa6d12daa76d42 WatchSource:0}: Error finding container bdc034ac178b7527774e421ed27c73b7ab52978e9f419f062bfa6d12daa76d42: Status 404 returned error can't find the container with id bdc034ac178b7527774e421ed27c73b7ab52978e9f419f062bfa6d12daa76d42
Jul 28 21:00:07.426109 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:07.426076    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8555f666d8-wmv8t" event={"ID":"336dff12-2e62-4970-89b7-6e58679fe739","Type":"ContainerStarted","Data":"bdc034ac178b7527774e421ed27c73b7ab52978e9f419f062bfa6d12daa76d42"}
Jul 28 21:00:09.437382 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:09.437342    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8555f666d8-wmv8t" event={"ID":"336dff12-2e62-4970-89b7-6e58679fe739","Type":"ContainerStarted","Data":"291f51d631ef7616b5f0d68d3b78bb5dcaaca89540e84518bec4be5199a884e2"}
Jul 28 21:00:09.437838 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:09.437414    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8555f666d8-wmv8t"
Jul 28 21:00:09.456684 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:09.456632    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8555f666d8-wmv8t" podStartSLOduration=1.953063841 podStartE2EDuration="3.456617835s" totalImagesPullingTime="1.503553994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:00:09.455886421 +0000 UTC m=+770.073468325" watchObservedRunningTime="2026-07-28 21:00:09.456617835 +0000 UTC m=+770.074199771"
Jul 28 21:00:20.443979 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:20.443950    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8555f666d8-wmv8t"
Jul 28 21:00:24.993450 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:24.993422    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:00:25.074876 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.072404    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8555f666d8-wmv8t"]
Jul 28 21:00:25.074876 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.072765    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8555f666d8-wmv8t" podUID="336dff12-2e62-4970-89b7-6e58679fe739" containerName="manager" containerID="cri-o://291f51d631ef7616b5f0d68d3b78bb5dcaaca89540e84518bec4be5199a884e2" gracePeriod=30
Jul 28 21:00:25.321044 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.321019    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8555f666d8-wmv8t"
Jul 28 21:00:25.410499 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.410474    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsf2d\" (UniqueName: \"kubernetes.io/projected/336dff12-2e62-4970-89b7-6e58679fe739-kube-api-access-zsf2d\") pod \"336dff12-2e62-4970-89b7-6e58679fe739\" (UID: \"336dff12-2e62-4970-89b7-6e58679fe739\") "
Jul 28 21:00:25.410682 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.410571    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/336dff12-2e62-4970-89b7-6e58679fe739-webhook-cert\") pod \"336dff12-2e62-4970-89b7-6e58679fe739\" (UID: \"336dff12-2e62-4970-89b7-6e58679fe739\") "
Jul 28 21:00:25.412521 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.412488    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336dff12-2e62-4970-89b7-6e58679fe739-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "336dff12-2e62-4970-89b7-6e58679fe739" (UID: "336dff12-2e62-4970-89b7-6e58679fe739"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:00:25.412521 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.412497    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336dff12-2e62-4970-89b7-6e58679fe739-kube-api-access-zsf2d" (OuterVolumeSpecName: "kube-api-access-zsf2d") pod "336dff12-2e62-4970-89b7-6e58679fe739" (UID: "336dff12-2e62-4970-89b7-6e58679fe739"). InnerVolumeSpecName "kube-api-access-zsf2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:00:25.511181 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.511152    2537 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/336dff12-2e62-4970-89b7-6e58679fe739-webhook-cert\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:00:25.511181 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.511180    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsf2d\" (UniqueName: \"kubernetes.io/projected/336dff12-2e62-4970-89b7-6e58679fe739-kube-api-access-zsf2d\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:00:25.511908 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.511882    2537 generic.go:356] "Generic (PLEG): container finished" podID="336dff12-2e62-4970-89b7-6e58679fe739" containerID="291f51d631ef7616b5f0d68d3b78bb5dcaaca89540e84518bec4be5199a884e2" exitCode=0
Jul 28 21:00:25.511975 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.511926    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8555f666d8-wmv8t"
Jul 28 21:00:25.512013 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.511972    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8555f666d8-wmv8t" event={"ID":"336dff12-2e62-4970-89b7-6e58679fe739","Type":"ContainerDied","Data":"291f51d631ef7616b5f0d68d3b78bb5dcaaca89540e84518bec4be5199a884e2"}
Jul 28 21:00:25.512088 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.512023    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8555f666d8-wmv8t" event={"ID":"336dff12-2e62-4970-89b7-6e58679fe739","Type":"ContainerDied","Data":"bdc034ac178b7527774e421ed27c73b7ab52978e9f419f062bfa6d12daa76d42"}
Jul 28 21:00:25.512088 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.512047    2537 scope.go:117] "RemoveContainer" containerID="291f51d631ef7616b5f0d68d3b78bb5dcaaca89540e84518bec4be5199a884e2"
Jul 28 21:00:25.520784 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.520766    2537 scope.go:117] "RemoveContainer" containerID="291f51d631ef7616b5f0d68d3b78bb5dcaaca89540e84518bec4be5199a884e2"
Jul 28 21:00:25.521087 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:00:25.521065    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291f51d631ef7616b5f0d68d3b78bb5dcaaca89540e84518bec4be5199a884e2\": container with ID starting with 291f51d631ef7616b5f0d68d3b78bb5dcaaca89540e84518bec4be5199a884e2 not found: ID does not exist" containerID="291f51d631ef7616b5f0d68d3b78bb5dcaaca89540e84518bec4be5199a884e2"
Jul 28 21:00:25.521152 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.521095    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291f51d631ef7616b5f0d68d3b78bb5dcaaca89540e84518bec4be5199a884e2"} err="failed to get container status \"291f51d631ef7616b5f0d68d3b78bb5dcaaca89540e84518bec4be5199a884e2\": rpc error: code = NotFound desc = could not find container \"291f51d631ef7616b5f0d68d3b78bb5dcaaca89540e84518bec4be5199a884e2\": container with ID starting with 291f51d631ef7616b5f0d68d3b78bb5dcaaca89540e84518bec4be5199a884e2 not found: ID does not exist"
Jul 28 21:00:25.535089 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.535068    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8555f666d8-wmv8t"]
Jul 28 21:00:25.539183 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:25.539163    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8555f666d8-wmv8t"]
Jul 28 21:00:26.020505 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.020474    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="336dff12-2e62-4970-89b7-6e58679fe739" path="/var/lib/kubelet/pods/336dff12-2e62-4970-89b7-6e58679fe739/volumes"
Jul 28 21:00:26.191831 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.191803    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:00:26.294334 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.294266    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8555f666d8-9vdt2"]
Jul 28 21:00:26.294647 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.294635    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="336dff12-2e62-4970-89b7-6e58679fe739" containerName="manager"
Jul 28 21:00:26.294689 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.294649    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="336dff12-2e62-4970-89b7-6e58679fe739" containerName="manager"
Jul 28 21:00:26.294724 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.294709    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="336dff12-2e62-4970-89b7-6e58679fe739" containerName="manager"
Jul 28 21:00:26.299099 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.299079    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8555f666d8-9vdt2"
Jul 28 21:00:26.302841 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.302816    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 21:00:26.302841 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.302834    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 21:00:26.303006 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.302822    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j4c6p\""
Jul 28 21:00:26.306483 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.306351    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8555f666d8-9vdt2"]
Jul 28 21:00:26.417971 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.417941    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1ec31bc-c698-416c-9bd4-f48888f77a9c-webhook-cert\") pod \"maas-controller-8555f666d8-9vdt2\" (UID: \"d1ec31bc-c698-416c-9bd4-f48888f77a9c\") " pod="opendatahub/maas-controller-8555f666d8-9vdt2"
Jul 28 21:00:26.418141 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.417982    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws6fz\" (UniqueName: \"kubernetes.io/projected/d1ec31bc-c698-416c-9bd4-f48888f77a9c-kube-api-access-ws6fz\") pod \"maas-controller-8555f666d8-9vdt2\" (UID: \"d1ec31bc-c698-416c-9bd4-f48888f77a9c\") " pod="opendatahub/maas-controller-8555f666d8-9vdt2"
Jul 28 21:00:26.518525 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.518493    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1ec31bc-c698-416c-9bd4-f48888f77a9c-webhook-cert\") pod \"maas-controller-8555f666d8-9vdt2\" (UID: \"d1ec31bc-c698-416c-9bd4-f48888f77a9c\") " pod="opendatahub/maas-controller-8555f666d8-9vdt2"
Jul 28 21:00:26.518692 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.518541    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws6fz\" (UniqueName: \"kubernetes.io/projected/d1ec31bc-c698-416c-9bd4-f48888f77a9c-kube-api-access-ws6fz\") pod \"maas-controller-8555f666d8-9vdt2\" (UID: \"d1ec31bc-c698-416c-9bd4-f48888f77a9c\") " pod="opendatahub/maas-controller-8555f666d8-9vdt2"
Jul 28 21:00:26.520914 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.520894    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1ec31bc-c698-416c-9bd4-f48888f77a9c-webhook-cert\") pod \"maas-controller-8555f666d8-9vdt2\" (UID: \"d1ec31bc-c698-416c-9bd4-f48888f77a9c\") " pod="opendatahub/maas-controller-8555f666d8-9vdt2"
Jul 28 21:00:26.527735 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.527712    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws6fz\" (UniqueName: \"kubernetes.io/projected/d1ec31bc-c698-416c-9bd4-f48888f77a9c-kube-api-access-ws6fz\") pod \"maas-controller-8555f666d8-9vdt2\" (UID: \"d1ec31bc-c698-416c-9bd4-f48888f77a9c\") " pod="opendatahub/maas-controller-8555f666d8-9vdt2"
Jul 28 21:00:26.610176 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.610156    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8555f666d8-9vdt2"
Jul 28 21:00:26.737434 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:26.737239    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8555f666d8-9vdt2"]
Jul 28 21:00:26.739807 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:00:26.739782    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ec31bc_c698_416c_9bd4_f48888f77a9c.slice/crio-fbef09baad5ab26fc970f2f38ef969c95f7a5f80b9cc78248260a6f0f6de4c83 WatchSource:0}: Error finding container fbef09baad5ab26fc970f2f38ef969c95f7a5f80b9cc78248260a6f0f6de4c83: Status 404 returned error can't find the container with id fbef09baad5ab26fc970f2f38ef969c95f7a5f80b9cc78248260a6f0f6de4c83
Jul 28 21:00:27.523220 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:27.523127    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8555f666d8-9vdt2" event={"ID":"d1ec31bc-c698-416c-9bd4-f48888f77a9c","Type":"ContainerStarted","Data":"411ab6c294640adcf6de7e42077f789645f86c2add27d323e318e8ffd59ec739"}
Jul 28 21:00:27.523220 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:27.523189    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8555f666d8-9vdt2" event={"ID":"d1ec31bc-c698-416c-9bd4-f48888f77a9c","Type":"ContainerStarted","Data":"fbef09baad5ab26fc970f2f38ef969c95f7a5f80b9cc78248260a6f0f6de4c83"}
Jul 28 21:00:27.523723 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:27.523245    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8555f666d8-9vdt2"
Jul 28 21:00:27.541200 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:27.541154    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8555f666d8-9vdt2" podStartSLOduration=1.215729201 podStartE2EDuration="1.541140518s" totalImagesPullingTime="325.411317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:00:27.540075317 +0000 UTC m=+788.157657218" watchObservedRunningTime="2026-07-28 21:00:27.541140518 +0000 UTC m=+788.158722419"
Jul 28 21:00:38.531070 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:00:38.531042    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8555f666d8-9vdt2"
Jul 28 21:01:07.589618 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:01:07.589581    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:07.688978 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:01:07.688949    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:07.789229 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:01:07.789201    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:13.756641 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:01:13.756606    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:14.292322 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:01:14.292287    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:14.393651 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:01:14.393615    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:19.855409 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:01:19.855379    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:19.912518 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:01:19.912487    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:02:19.863567 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:02:19.863492    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:02:19.877915 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:02:19.877889    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:03:01.016456 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:03:01.016425    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:03:01.104068 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:03:01.104024    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:03:25.872004 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:03:25.871964    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:03:37.971383 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:03:37.971348    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:43.182766 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:05:43.182713    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:07:19.915576 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:07:19.915472    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:07:19.919391 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:07:19.918453    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:09:32.481898 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.481863    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"]
Jul 28 21:09:32.485290 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.485273    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.488132 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.488111    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3022b1f1-kserve-self-signed-certs\""
Jul 28 21:09:32.505153 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.505129    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"]
Jul 28 21:09:32.531369 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.531339    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-tmp-dir\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.531505 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.531390    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38617ce6-3a4d-4873-a5df-92ed6170f33c-tls-certs\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.531505 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.531437    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh84x\" (UniqueName: \"kubernetes.io/projected/38617ce6-3a4d-4873-a5df-92ed6170f33c-kube-api-access-xh84x\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.531505 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.531465    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-dshm\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.531671 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.531539    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-home\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.531671 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.531607    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-model-cache\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.632106 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.632070    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-home\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.632106 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.632091    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:09:32.632334 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.632137    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-model-cache\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.632334 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.632187    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-tmp-dir\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.632334 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.632233    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38617ce6-3a4d-4873-a5df-92ed6170f33c-tls-certs\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.632334 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.632277    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh84x\" (UniqueName: \"kubernetes.io/projected/38617ce6-3a4d-4873-a5df-92ed6170f33c-kube-api-access-xh84x\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.632334 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.632312    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-dshm\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.632603 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.632427    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-home\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.632603 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.632503    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-model-cache\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.632603 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.632596    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-tmp-dir\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.634611 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.634590    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-dshm\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.634751 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.634730    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38617ce6-3a4d-4873-a5df-92ed6170f33c-tls-certs\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.641057 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.641037    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh84x\" (UniqueName: \"kubernetes.io/projected/38617ce6-3a4d-4873-a5df-92ed6170f33c-kube-api-access-xh84x\") pod \"e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") " pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.794111 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.794031    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:32.932163 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:32.932138    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"]
Jul 28 21:09:32.934331 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:09:32.934295    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38617ce6_3a4d_4873_a5df_92ed6170f33c.slice/crio-809ce61778238cf8ef4abe7ff891ea4b6a5bd1acfab9a0939092111c9ca87f4e WatchSource:0}: Error finding container 809ce61778238cf8ef4abe7ff891ea4b6a5bd1acfab9a0939092111c9ca87f4e: Status 404 returned error can't find the container with id 809ce61778238cf8ef4abe7ff891ea4b6a5bd1acfab9a0939092111c9ca87f4e
Jul 28 21:09:33.200395 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:33.200364    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft" event={"ID":"38617ce6-3a4d-4873-a5df-92ed6170f33c","Type":"ContainerStarted","Data":"8e69b27c1f152b4a5e4c597010d5fd4fde148727c0263b9ecddeffaa6d4fd244"}
Jul 28 21:09:33.200395 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:33.200407    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft" event={"ID":"38617ce6-3a4d-4873-a5df-92ed6170f33c","Type":"ContainerStarted","Data":"809ce61778238cf8ef4abe7ff891ea4b6a5bd1acfab9a0939092111c9ca87f4e"}
Jul 28 21:09:33.218779 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:33.218706    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft" podStartSLOduration=1.218691875 podStartE2EDuration="1.218691875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:09:33.217594183 +0000 UTC m=+1333.835176086" watchObservedRunningTime="2026-07-28 21:09:33.218691875 +0000 UTC m=+1333.836273777"
Jul 28 21:09:42.794749 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:42.794661    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:42.795231 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:42.794812    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:42.807183 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:42.807156    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:43.263169 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:43.263139    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:09:50.462305 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.462267    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"]
Jul 28 21:09:50.466421 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.466396    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.469314 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.469293    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-3022b1f1-kserve-self-signed-certs\""
Jul 28 21:09:50.475020 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.474992    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"]
Jul 28 21:09:50.498141 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.498115    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:09:50.593646 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.593617    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-dshm\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.593790 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.593682    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-tmp-dir\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.593790 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.593714    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-home\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.593790 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.593743    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6cd5\" (UniqueName: \"kubernetes.io/projected/e73ff4c5-63bd-40a5-9785-aa866d44a560-kube-api-access-q6cd5\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.593790 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.593768    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-model-cache\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.594040 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.593842    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73ff4c5-63bd-40a5-9785-aa866d44a560-tls-certs\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.694795 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.694752    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73ff4c5-63bd-40a5-9785-aa866d44a560-tls-certs\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.694980 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.694828    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-dshm\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.694980 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.694930    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-tmp-dir\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.694980 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.694964    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-home\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.695200 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.695003    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6cd5\" (UniqueName: \"kubernetes.io/projected/e73ff4c5-63bd-40a5-9785-aa866d44a560-kube-api-access-q6cd5\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.695200 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.695035    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-model-cache\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.695310 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.695285    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-tmp-dir\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.695372 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.695316    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-model-cache\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.695429 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.695375    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-home\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.697119 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.697094    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-dshm\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.697287 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.697270    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73ff4c5-63bd-40a5-9785-aa866d44a560-tls-certs\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.704517 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.704499    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6cd5\" (UniqueName: \"kubernetes.io/projected/e73ff4c5-63bd-40a5-9785-aa866d44a560-kube-api-access-q6cd5\") pod \"e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") " pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.778008 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.777948    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:09:50.912343 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:50.912284    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"]
Jul 28 21:09:50.914697 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:09:50.914668    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73ff4c5_63bd_40a5_9785_aa866d44a560.slice/crio-90208ff811e4fc95dad1487c766e9a0dd7b64dc91e37aad44e6463e81b10f0fa WatchSource:0}: Error finding container 90208ff811e4fc95dad1487c766e9a0dd7b64dc91e37aad44e6463e81b10f0fa: Status 404 returned error can't find the container with id 90208ff811e4fc95dad1487c766e9a0dd7b64dc91e37aad44e6463e81b10f0fa
Jul 28 21:09:51.290189 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:51.290151    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws" event={"ID":"e73ff4c5-63bd-40a5-9785-aa866d44a560","Type":"ContainerStarted","Data":"342946c1416695240c2b71342f6d0dd1927910472f43766ea41b2cb59166380a"}
Jul 28 21:09:51.290189 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:51.290192    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws" event={"ID":"e73ff4c5-63bd-40a5-9785-aa866d44a560","Type":"ContainerStarted","Data":"90208ff811e4fc95dad1487c766e9a0dd7b64dc91e37aad44e6463e81b10f0fa"}
Jul 28 21:09:51.307960 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:51.307912    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws" podStartSLOduration=1.30789627 podStartE2EDuration="1.30789627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:09:51.307278866 +0000 UTC m=+1351.924860769" watchObservedRunningTime="2026-07-28 21:09:51.30789627 +0000 UTC m=+1351.925478190"
Jul 28 21:09:55.376028 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:55.375989    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:09:55.476549 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:09:55.476519    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:10:00.778570 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:00.778516    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:10:00.779085 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:00.778595    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:10:00.791027 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:00.791005    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:10:01.353108 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:01.353079    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:10:27.939838 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:27.939803    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"]
Jul 28 21:10:27.949079 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:27.949048    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:10:28.906692 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:28.906650    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"]
Jul 28 21:10:28.921027 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:28.920999    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:10:42.966891 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:42.966809    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft" podUID="38617ce6-3a4d-4873-a5df-92ed6170f33c" containerName="main" containerID="cri-o://8e69b27c1f152b4a5e4c597010d5fd4fde148727c0263b9ecddeffaa6d4fd244" gracePeriod=45
Jul 28 21:10:43.222010 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.221945    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:10:43.351734 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.351702    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38617ce6-3a4d-4873-a5df-92ed6170f33c-tls-certs\") pod \"38617ce6-3a4d-4873-a5df-92ed6170f33c\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") "
Jul 28 21:10:43.351734 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.351739    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-dshm\") pod \"38617ce6-3a4d-4873-a5df-92ed6170f33c\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") "
Jul 28 21:10:43.351981 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.351774    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh84x\" (UniqueName: \"kubernetes.io/projected/38617ce6-3a4d-4873-a5df-92ed6170f33c-kube-api-access-xh84x\") pod \"38617ce6-3a4d-4873-a5df-92ed6170f33c\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") "
Jul 28 21:10:43.351981 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.351843    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-home\") pod \"38617ce6-3a4d-4873-a5df-92ed6170f33c\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") "
Jul 28 21:10:43.351981 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.351889    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-tmp-dir\") pod \"38617ce6-3a4d-4873-a5df-92ed6170f33c\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") "
Jul 28 21:10:43.351981 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.351951    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-model-cache\") pod \"38617ce6-3a4d-4873-a5df-92ed6170f33c\" (UID: \"38617ce6-3a4d-4873-a5df-92ed6170f33c\") "
Jul 28 21:10:43.352191 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.352140    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-home" (OuterVolumeSpecName: "home") pod "38617ce6-3a4d-4873-a5df-92ed6170f33c" (UID: "38617ce6-3a4d-4873-a5df-92ed6170f33c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:10:43.352247 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.352180    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38617ce6-3a4d-4873-a5df-92ed6170f33c" (UID: "38617ce6-3a4d-4873-a5df-92ed6170f33c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:10:43.352247 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.352227    2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-home\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:10:43.352400 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.352377    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-model-cache" (OuterVolumeSpecName: "model-cache") pod "38617ce6-3a4d-4873-a5df-92ed6170f33c" (UID: "38617ce6-3a4d-4873-a5df-92ed6170f33c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:10:43.354071 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.354051    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38617ce6-3a4d-4873-a5df-92ed6170f33c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38617ce6-3a4d-4873-a5df-92ed6170f33c" (UID: "38617ce6-3a4d-4873-a5df-92ed6170f33c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:10:43.354326 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.354304    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-dshm" (OuterVolumeSpecName: "dshm") pod "38617ce6-3a4d-4873-a5df-92ed6170f33c" (UID: "38617ce6-3a4d-4873-a5df-92ed6170f33c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:10:43.354425 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.354337    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38617ce6-3a4d-4873-a5df-92ed6170f33c-kube-api-access-xh84x" (OuterVolumeSpecName: "kube-api-access-xh84x") pod "38617ce6-3a4d-4873-a5df-92ed6170f33c" (UID: "38617ce6-3a4d-4873-a5df-92ed6170f33c"). InnerVolumeSpecName "kube-api-access-xh84x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:10:43.453264 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.453235    2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38617ce6-3a4d-4873-a5df-92ed6170f33c-tls-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:10:43.453264 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.453264    2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-dshm\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:10:43.453453 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.453274    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh84x\" (UniqueName: \"kubernetes.io/projected/38617ce6-3a4d-4873-a5df-92ed6170f33c-kube-api-access-xh84x\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:10:43.453453 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.453285    2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-tmp-dir\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:10:43.453453 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.453296    2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38617ce6-3a4d-4873-a5df-92ed6170f33c-model-cache\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:10:43.550181 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.550086    2537 generic.go:356] "Generic (PLEG): container finished" podID="38617ce6-3a4d-4873-a5df-92ed6170f33c" containerID="8e69b27c1f152b4a5e4c597010d5fd4fde148727c0263b9ecddeffaa6d4fd244" exitCode=0
Jul 28 21:10:43.550181 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.550126    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"
Jul 28 21:10:43.550409 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.550178    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft" event={"ID":"38617ce6-3a4d-4873-a5df-92ed6170f33c","Type":"ContainerDied","Data":"8e69b27c1f152b4a5e4c597010d5fd4fde148727c0263b9ecddeffaa6d4fd244"}
Jul 28 21:10:43.550409 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.550226    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft" event={"ID":"38617ce6-3a4d-4873-a5df-92ed6170f33c","Type":"ContainerDied","Data":"809ce61778238cf8ef4abe7ff891ea4b6a5bd1acfab9a0939092111c9ca87f4e"}
Jul 28 21:10:43.550409 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.550247    2537 scope.go:117] "RemoveContainer" containerID="8e69b27c1f152b4a5e4c597010d5fd4fde148727c0263b9ecddeffaa6d4fd244"
Jul 28 21:10:43.559788 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.559773    2537 scope.go:117] "RemoveContainer" containerID="8e69b27c1f152b4a5e4c597010d5fd4fde148727c0263b9ecddeffaa6d4fd244"
Jul 28 21:10:43.560068 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:10:43.560045    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e69b27c1f152b4a5e4c597010d5fd4fde148727c0263b9ecddeffaa6d4fd244\": container with ID starting with 8e69b27c1f152b4a5e4c597010d5fd4fde148727c0263b9ecddeffaa6d4fd244 not found: ID does not exist" containerID="8e69b27c1f152b4a5e4c597010d5fd4fde148727c0263b9ecddeffaa6d4fd244"
Jul 28 21:10:43.560141 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.560076    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e69b27c1f152b4a5e4c597010d5fd4fde148727c0263b9ecddeffaa6d4fd244"} err="failed to get container status \"8e69b27c1f152b4a5e4c597010d5fd4fde148727c0263b9ecddeffaa6d4fd244\": rpc error: code = NotFound desc = could not find container \"8e69b27c1f152b4a5e4c597010d5fd4fde148727c0263b9ecddeffaa6d4fd244\": container with ID starting with 8e69b27c1f152b4a5e4c597010d5fd4fde148727c0263b9ecddeffaa6d4fd244 not found: ID does not exist"
Jul 28 21:10:43.572103 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.572079    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"]
Jul 28 21:10:43.575498 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.575476    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3022b1f1-kserve-587c476d8c-tbxft"]
Jul 28 21:10:43.925956 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:43.925889    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws" podUID="e73ff4c5-63bd-40a5-9785-aa866d44a560" containerName="main" containerID="cri-o://342946c1416695240c2b71342f6d0dd1927910472f43766ea41b2cb59166380a" gracePeriod=45
Jul 28 21:10:44.023971 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.023929    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38617ce6-3a4d-4873-a5df-92ed6170f33c" path="/var/lib/kubelet/pods/38617ce6-3a4d-4873-a5df-92ed6170f33c/volumes"
Jul 28 21:10:44.172259 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.172234    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:10:44.260897 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.260796    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-home\") pod \"e73ff4c5-63bd-40a5-9785-aa866d44a560\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") "
Jul 28 21:10:44.261036 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.260903    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-model-cache\") pod \"e73ff4c5-63bd-40a5-9785-aa866d44a560\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") "
Jul 28 21:10:44.261036 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.260962    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73ff4c5-63bd-40a5-9785-aa866d44a560-tls-certs\") pod \"e73ff4c5-63bd-40a5-9785-aa866d44a560\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") "
Jul 28 21:10:44.261036 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.260990    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-tmp-dir\") pod \"e73ff4c5-63bd-40a5-9785-aa866d44a560\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") "
Jul 28 21:10:44.261213 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.261081    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-dshm\") pod \"e73ff4c5-63bd-40a5-9785-aa866d44a560\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") "
Jul 28 21:10:44.261213 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.261097    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-home" (OuterVolumeSpecName: "home") pod "e73ff4c5-63bd-40a5-9785-aa866d44a560" (UID: "e73ff4c5-63bd-40a5-9785-aa866d44a560"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:10:44.261213 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.261107    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6cd5\" (UniqueName: \"kubernetes.io/projected/e73ff4c5-63bd-40a5-9785-aa866d44a560-kube-api-access-q6cd5\") pod \"e73ff4c5-63bd-40a5-9785-aa866d44a560\" (UID: \"e73ff4c5-63bd-40a5-9785-aa866d44a560\") "
Jul 28 21:10:44.261213 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.261134    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-model-cache" (OuterVolumeSpecName: "model-cache") pod "e73ff4c5-63bd-40a5-9785-aa866d44a560" (UID: "e73ff4c5-63bd-40a5-9785-aa866d44a560"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:10:44.261421 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.261310    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e73ff4c5-63bd-40a5-9785-aa866d44a560" (UID: "e73ff4c5-63bd-40a5-9785-aa866d44a560"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:10:44.261421 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.261398    2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-home\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:10:44.261421 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.261415    2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-model-cache\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:10:44.261520 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.261426    2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-tmp-dir\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:10:44.263194 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.263172    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-dshm" (OuterVolumeSpecName: "dshm") pod "e73ff4c5-63bd-40a5-9785-aa866d44a560" (UID: "e73ff4c5-63bd-40a5-9785-aa866d44a560"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:10:44.263298 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.263235    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e73ff4c5-63bd-40a5-9785-aa866d44a560-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e73ff4c5-63bd-40a5-9785-aa866d44a560" (UID: "e73ff4c5-63bd-40a5-9785-aa866d44a560"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:10:44.263298 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.263266    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73ff4c5-63bd-40a5-9785-aa866d44a560-kube-api-access-q6cd5" (OuterVolumeSpecName: "kube-api-access-q6cd5") pod "e73ff4c5-63bd-40a5-9785-aa866d44a560" (UID: "e73ff4c5-63bd-40a5-9785-aa866d44a560"). InnerVolumeSpecName "kube-api-access-q6cd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:10:44.362550 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.362524    2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73ff4c5-63bd-40a5-9785-aa866d44a560-dshm\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:10:44.362550 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.362549    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6cd5\" (UniqueName: \"kubernetes.io/projected/e73ff4c5-63bd-40a5-9785-aa866d44a560-kube-api-access-q6cd5\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:10:44.362713 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.362560    2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73ff4c5-63bd-40a5-9785-aa866d44a560-tls-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:10:44.557284 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.557195    2537 generic.go:356] "Generic (PLEG): container finished" podID="e73ff4c5-63bd-40a5-9785-aa866d44a560" containerID="342946c1416695240c2b71342f6d0dd1927910472f43766ea41b2cb59166380a" exitCode=0
Jul 28 21:10:44.557284 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.557238    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"
Jul 28 21:10:44.557504 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.557281    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws" event={"ID":"e73ff4c5-63bd-40a5-9785-aa866d44a560","Type":"ContainerDied","Data":"342946c1416695240c2b71342f6d0dd1927910472f43766ea41b2cb59166380a"}
Jul 28 21:10:44.557504 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.557323    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws" event={"ID":"e73ff4c5-63bd-40a5-9785-aa866d44a560","Type":"ContainerDied","Data":"90208ff811e4fc95dad1487c766e9a0dd7b64dc91e37aad44e6463e81b10f0fa"}
Jul 28 21:10:44.557504 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.557340    2537 scope.go:117] "RemoveContainer" containerID="342946c1416695240c2b71342f6d0dd1927910472f43766ea41b2cb59166380a"
Jul 28 21:10:44.566596 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.566580    2537 scope.go:117] "RemoveContainer" containerID="342946c1416695240c2b71342f6d0dd1927910472f43766ea41b2cb59166380a"
Jul 28 21:10:44.566849 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:10:44.566831    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342946c1416695240c2b71342f6d0dd1927910472f43766ea41b2cb59166380a\": container with ID starting with 342946c1416695240c2b71342f6d0dd1927910472f43766ea41b2cb59166380a not found: ID does not exist" containerID="342946c1416695240c2b71342f6d0dd1927910472f43766ea41b2cb59166380a"
Jul 28 21:10:44.566929 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.566872    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342946c1416695240c2b71342f6d0dd1927910472f43766ea41b2cb59166380a"} err="failed to get container status \"342946c1416695240c2b71342f6d0dd1927910472f43766ea41b2cb59166380a\": rpc error: code = NotFound desc = could not find container \"342946c1416695240c2b71342f6d0dd1927910472f43766ea41b2cb59166380a\": container with ID starting with 342946c1416695240c2b71342f6d0dd1927910472f43766ea41b2cb59166380a not found: ID does not exist"
Jul 28 21:10:44.578376 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.578353    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"]
Jul 28 21:10:44.584196 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:44.584177    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-3022b1f1-kserve-5cc69f6fcf-54jws"]
Jul 28 21:10:46.021408 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:10:46.021372    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e73ff4c5-63bd-40a5-9785-aa866d44a560" path="/var/lib/kubelet/pods/e73ff4c5-63bd-40a5-9785-aa866d44a560/volumes"
Jul 28 21:12:19.946087 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:12:19.945969    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:12:19.947158 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:12:19.946516    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:12:25.330154 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:12:25.330116    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:12:44.898973 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:12:44.898935    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:12:44.955973 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:12:44.955942    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:13:33.408994 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:13:33.408950    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:13:53.044728 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:13:53.044687    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:13:53.348395 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:13:53.348344    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:13:55.986737 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:13:55.986694    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:13:56.082157 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:13:56.082117    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:13:56.415637 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:13:56.415599    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:02.282935 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.282901    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:02.383199 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.383162    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:02.509760 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.509723    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:02.778674 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.778645    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"]
Jul 28 21:14:02.779049 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.779036    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e73ff4c5-63bd-40a5-9785-aa866d44a560" containerName="main"
Jul 28 21:14:02.779049 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.779050    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="e73ff4c5-63bd-40a5-9785-aa866d44a560" containerName="main"
Jul 28 21:14:02.779174 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.779068    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38617ce6-3a4d-4873-a5df-92ed6170f33c" containerName="main"
Jul 28 21:14:02.779174 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.779074    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="38617ce6-3a4d-4873-a5df-92ed6170f33c" containerName="main"
Jul 28 21:14:02.779174 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.779132    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="e73ff4c5-63bd-40a5-9785-aa866d44a560" containerName="main"
Jul 28 21:14:02.779174 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.779139    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="38617ce6-3a4d-4873-a5df-92ed6170f33c" containerName="main"
Jul 28 21:14:02.782242 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.782228    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.785005 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.784984    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a4f1163e-openshift-default-dockercfg-cq9qr\""
Jul 28 21:14:02.803024 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.803002    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"]
Jul 28 21:14:02.807750 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.807726    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:02.829733 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.829706    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-credential-socket\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.829884 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.829748    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-istio-data\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.829884 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.829779    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-workload-socket\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.829884 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.829820    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-istio-envoy\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.829884 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.829840    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a94bf216-c711-488c-b443-e288a7c0be87-istiod-ca-cert\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.830102 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.829894    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-workload-certs\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.830102 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.829909    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b8ff\" (UniqueName: \"kubernetes.io/projected/a94bf216-c711-488c-b443-e288a7c0be87-kube-api-access-9b8ff\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.830102 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.830034    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a94bf216-c711-488c-b443-e288a7c0be87-istio-token\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.830102 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.830088    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a94bf216-c711-488c-b443-e288a7c0be87-istio-podinfo\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.930923 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.930888    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-workload-certs\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.931129 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.930929    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b8ff\" (UniqueName: \"kubernetes.io/projected/a94bf216-c711-488c-b443-e288a7c0be87-kube-api-access-9b8ff\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.931129 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.930994    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a94bf216-c711-488c-b443-e288a7c0be87-istio-token\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.931129 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.931025    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a94bf216-c711-488c-b443-e288a7c0be87-istio-podinfo\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.931129 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.931059    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-credential-socket\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.931129 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.931109    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-istio-data\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.931403 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.931147    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-workload-socket\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.931403 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.931196    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-istio-envoy\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.931403 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.931228    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a94bf216-c711-488c-b443-e288a7c0be87-istiod-ca-cert\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.931564 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.931436    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-credential-socket\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.931564 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.931464    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-istio-data\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.931564 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.931536    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-workload-socket\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.931564 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.931544    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-workload-certs\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.931976 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.931955    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a94bf216-c711-488c-b443-e288a7c0be87-istiod-ca-cert\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.933379 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.933349    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-istio-envoy\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.933498 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.933454    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a94bf216-c711-488c-b443-e288a7c0be87-istio-podinfo\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.939696 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.939674    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a94bf216-c711-488c-b443-e288a7c0be87-istio-token\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:02.940312 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:02.940295    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b8ff\" (UniqueName: \"kubernetes.io/projected/a94bf216-c711-488c-b443-e288a7c0be87-kube-api-access-9b8ff\") pod \"e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") " pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:03.068046 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:03.068007    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:03.092907 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:03.092878    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:03.234572 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:03.234542    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"]
Jul 28 21:14:03.237121 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:14:03.237086    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94bf216_c711_488c_b443_e288a7c0be87.slice/crio-e3e8116ef00a7335513831beada88becffce384a61c49cc9c47dbf0d6f1ed6a8 WatchSource:0}: Error finding container e3e8116ef00a7335513831beada88becffce384a61c49cc9c47dbf0d6f1ed6a8: Status 404 returned error can't find the container with id e3e8116ef00a7335513831beada88becffce384a61c49cc9c47dbf0d6f1ed6a8
Jul 28 21:14:03.239380 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:03.239346    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:14:03.239465 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:03.239417    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:14:03.239465 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:03.239459    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:14:03.533326 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:03.533237    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg" event={"ID":"a94bf216-c711-488c-b443-e288a7c0be87","Type":"ContainerStarted","Data":"d3b33a1d975dd5fd9f3f7defb8267cbe266a7e720b878fbc98460216f678e292"}
Jul 28 21:14:03.533326 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:03.533298    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg" event={"ID":"a94bf216-c711-488c-b443-e288a7c0be87","Type":"ContainerStarted","Data":"e3e8116ef00a7335513831beada88becffce384a61c49cc9c47dbf0d6f1ed6a8"}
Jul 28 21:14:03.554474 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:03.554420    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg" podStartSLOduration=1.554401425 podStartE2EDuration="1.554401425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:14:03.55377313 +0000 UTC m=+1604.171355033" watchObservedRunningTime="2026-07-28 21:14:03.554401425 +0000 UTC m=+1604.171983328"
Jul 28 21:14:04.093315 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:04.093283    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:04.098134 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:04.098109    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:04.538445 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:04.538373    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:04.539246 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:04.539226    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:04.880909 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:04.880870    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:04.959997 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:04.959966    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"]
Jul 28 21:14:04.961085 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:04.961052    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:04.963625 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:04.963605    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:04.966151 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:04.966131    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a4f1163e-serving-cert\""
Jul 28 21:14:04.971453 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:04.971434    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"]
Jul 28 21:14:05.024640 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:05.024610    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:05.053276 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:05.053250    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0472bae2-b64e-457b-bc38-8da3168cb6e3-maas-api-tls\") pod \"maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr\" (UID: \"0472bae2-b64e-457b-bc38-8da3168cb6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:05.053410 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:05.053287    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0472bae2-b64e-457b-bc38-8da3168cb6e3-openshift-service-ca\") pod \"maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr\" (UID: \"0472bae2-b64e-457b-bc38-8da3168cb6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:05.053410 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:05.053308    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h69v7\" (UniqueName: \"kubernetes.io/projected/0472bae2-b64e-457b-bc38-8da3168cb6e3-kube-api-access-h69v7\") pod \"maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr\" (UID: \"0472bae2-b64e-457b-bc38-8da3168cb6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:05.154317 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:05.154226    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0472bae2-b64e-457b-bc38-8da3168cb6e3-maas-api-tls\") pod \"maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr\" (UID: \"0472bae2-b64e-457b-bc38-8da3168cb6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:05.154317 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:05.154266    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0472bae2-b64e-457b-bc38-8da3168cb6e3-openshift-service-ca\") pod \"maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr\" (UID: \"0472bae2-b64e-457b-bc38-8da3168cb6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:05.154536 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:14:05.154381    2537 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-serving-cert: secret "maas-api-e2e-ait-a4f1163e-serving-cert" not found
Jul 28 21:14:05.154536 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:14:05.154444    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0472bae2-b64e-457b-bc38-8da3168cb6e3-maas-api-tls podName:0472bae2-b64e-457b-bc38-8da3168cb6e3 nodeName:}" failed. No retries permitted until 2026-07-28 21:14:05.654426556 +0000 UTC m=+1606.272008435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0472bae2-b64e-457b-bc38-8da3168cb6e3-maas-api-tls") pod "maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr" (UID: "0472bae2-b64e-457b-bc38-8da3168cb6e3") : secret "maas-api-e2e-ait-a4f1163e-serving-cert" not found
Jul 28 21:14:05.154663 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:05.154532    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h69v7\" (UniqueName: \"kubernetes.io/projected/0472bae2-b64e-457b-bc38-8da3168cb6e3-kube-api-access-h69v7\") pod \"maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr\" (UID: \"0472bae2-b64e-457b-bc38-8da3168cb6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:05.155045 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:05.155023    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0472bae2-b64e-457b-bc38-8da3168cb6e3-openshift-service-ca\") pod \"maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr\" (UID: \"0472bae2-b64e-457b-bc38-8da3168cb6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:05.164897 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:05.164877    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h69v7\" (UniqueName: \"kubernetes.io/projected/0472bae2-b64e-457b-bc38-8da3168cb6e3-kube-api-access-h69v7\") pod \"maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr\" (UID: \"0472bae2-b64e-457b-bc38-8da3168cb6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:05.424012 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:05.423880    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:05.557721 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:05.557687    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:05.658846 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:05.658812    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0472bae2-b64e-457b-bc38-8da3168cb6e3-maas-api-tls\") pod \"maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr\" (UID: \"0472bae2-b64e-457b-bc38-8da3168cb6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:05.661530 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:05.661505    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0472bae2-b64e-457b-bc38-8da3168cb6e3-maas-api-tls\") pod \"maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr\" (UID: \"0472bae2-b64e-457b-bc38-8da3168cb6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:05.665320 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:05.665292    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:05.877199 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:05.877165    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:06.009062 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:06.009028    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"]
Jul 28 21:14:06.012171 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:14:06.012137    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0472bae2_b64e_457b_bc38_8da3168cb6e3.slice/crio-1029111bebf303984fe6539aa72b522f0a11ab4d2b13f4a202fed385434dc95f WatchSource:0}: Error finding container 1029111bebf303984fe6539aa72b522f0a11ab4d2b13f4a202fed385434dc95f: Status 404 returned error can't find the container with id 1029111bebf303984fe6539aa72b522f0a11ab4d2b13f4a202fed385434dc95f
Jul 28 21:14:06.013893 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:06.013873    2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 21:14:06.549521 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:06.549485    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr" event={"ID":"0472bae2-b64e-457b-bc38-8da3168cb6e3","Type":"ContainerStarted","Data":"1029111bebf303984fe6539aa72b522f0a11ab4d2b13f4a202fed385434dc95f"}
Jul 28 21:14:07.556122 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:07.556042    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr" event={"ID":"0472bae2-b64e-457b-bc38-8da3168cb6e3","Type":"ContainerStarted","Data":"15fc93412c2e717c24642b053100aa09dc97f1613daaa4d17e4e467e055158b0"}
Jul 28 21:14:07.556122 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:07.556112    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:07.574387 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:07.574336    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr" podStartSLOduration=2.399913353 podStartE2EDuration="3.574321252s" totalImagesPullingTime="1.174407899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:14:07.572876337 +0000 UTC m=+1608.190458241" watchObservedRunningTime="2026-07-28 21:14:07.574321252 +0000 UTC m=+1608.191903154"
Jul 28 21:14:13.567589 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:13.567554    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:15.597916 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:15.597824    2537 generic.go:356] "Generic (PLEG): container finished" podID="a94bf216-c711-488c-b443-e288a7c0be87" containerID="d3b33a1d975dd5fd9f3f7defb8267cbe266a7e720b878fbc98460216f678e292" exitCode=0
Jul 28 21:14:15.597916 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:15.597896    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg" event={"ID":"a94bf216-c711-488c-b443-e288a7c0be87","Type":"ContainerDied","Data":"d3b33a1d975dd5fd9f3f7defb8267cbe266a7e720b878fbc98460216f678e292"}
Jul 28 21:14:15.598358 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:15.598284    2537 scope.go:117] "RemoveContainer" containerID="d3b33a1d975dd5fd9f3f7defb8267cbe266a7e720b878fbc98460216f678e292"
Jul 28 21:14:15.599264 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:15.599235    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:14:15.599315 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:15.599306    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:14:15.599351 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:15.599337    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:14:16.604536 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:16.604499    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg" event={"ID":"a94bf216-c711-488c-b443-e288a7c0be87","Type":"ContainerStarted","Data":"d9312a85a7a957d8810c2e85ba310d474499786a20525e52e23e63c1f44afd7a"}
Jul 28 21:14:17.093375 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:17.093341    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:17.094600 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:17.094565    2537 patch_prober.go:28] interesting pod/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 28 21:14:17.094723 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:17.094624    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg" podUID="a94bf216-c711-488c-b443-e288a7c0be87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 28 21:14:18.093850 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:18.093814    2537 patch_prober.go:28] interesting pod/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 28 21:14:18.094275 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:18.093892    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg" podUID="a94bf216-c711-488c-b443-e288a7c0be87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 28 21:14:19.093401 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:19.093369    2537 patch_prober.go:28] interesting pod/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 28 21:14:19.093584 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:19.093420    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg" podUID="a94bf216-c711-488c-b443-e288a7c0be87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 28 21:14:19.539018 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:19.538934    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:20.094235 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:20.094195    2537 patch_prober.go:28] interesting pod/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 28 21:14:20.094427 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:20.094273    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg" podUID="a94bf216-c711-488c-b443-e288a7c0be87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 28 21:14:21.097342 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:21.097313    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:21.098247 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:21.098228    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:35.632073 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:35.632036    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"]
Jul 28 21:14:35.632723 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:35.632313    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr" podUID="0472bae2-b64e-457b-bc38-8da3168cb6e3" containerName="maas-api" containerID="cri-o://15fc93412c2e717c24642b053100aa09dc97f1613daaa4d17e4e467e055158b0" gracePeriod=30
Jul 28 21:14:35.658447 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:35.658420    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:35.898091 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:35.898027    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:35.947473 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:35.947443    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"]
Jul 28 21:14:35.947699 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:35.947676    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg" podUID="a94bf216-c711-488c-b443-e288a7c0be87" containerName="istio-proxy" containerID="cri-o://d9312a85a7a957d8810c2e85ba310d474499786a20525e52e23e63c1f44afd7a" gracePeriod=30
Jul 28 21:14:36.040481 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.040447    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0472bae2-b64e-457b-bc38-8da3168cb6e3-openshift-service-ca\") pod \"0472bae2-b64e-457b-bc38-8da3168cb6e3\" (UID: \"0472bae2-b64e-457b-bc38-8da3168cb6e3\") "
Jul 28 21:14:36.040642 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.040507    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h69v7\" (UniqueName: \"kubernetes.io/projected/0472bae2-b64e-457b-bc38-8da3168cb6e3-kube-api-access-h69v7\") pod \"0472bae2-b64e-457b-bc38-8da3168cb6e3\" (UID: \"0472bae2-b64e-457b-bc38-8da3168cb6e3\") "
Jul 28 21:14:36.040740 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.040718    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0472bae2-b64e-457b-bc38-8da3168cb6e3-maas-api-tls\") pod \"0472bae2-b64e-457b-bc38-8da3168cb6e3\" (UID: \"0472bae2-b64e-457b-bc38-8da3168cb6e3\") "
Jul 28 21:14:36.040984 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.040958    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0472bae2-b64e-457b-bc38-8da3168cb6e3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0472bae2-b64e-457b-bc38-8da3168cb6e3" (UID: "0472bae2-b64e-457b-bc38-8da3168cb6e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:14:36.041114 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.041059    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0472bae2-b64e-457b-bc38-8da3168cb6e3-openshift-service-ca\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:14:36.042642 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.042611    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0472bae2-b64e-457b-bc38-8da3168cb6e3-kube-api-access-h69v7" (OuterVolumeSpecName: "kube-api-access-h69v7") pod "0472bae2-b64e-457b-bc38-8da3168cb6e3" (UID: "0472bae2-b64e-457b-bc38-8da3168cb6e3"). InnerVolumeSpecName "kube-api-access-h69v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:14:36.042737 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.042692    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0472bae2-b64e-457b-bc38-8da3168cb6e3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0472bae2-b64e-457b-bc38-8da3168cb6e3" (UID: "0472bae2-b64e-457b-bc38-8da3168cb6e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:14:36.100056 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.100024    2537 patch_prober.go:28] interesting pod/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 21:14:36.100203 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.100090    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg" podUID="a94bf216-c711-488c-b443-e288a7c0be87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 21:14:36.142246 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.142125    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h69v7\" (UniqueName: \"kubernetes.io/projected/0472bae2-b64e-457b-bc38-8da3168cb6e3-kube-api-access-h69v7\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:14:36.142246 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.142155    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0472bae2-b64e-457b-bc38-8da3168cb6e3-maas-api-tls\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:14:36.718717 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.718682    2537 generic.go:356] "Generic (PLEG): container finished" podID="0472bae2-b64e-457b-bc38-8da3168cb6e3" containerID="15fc93412c2e717c24642b053100aa09dc97f1613daaa4d17e4e467e055158b0" exitCode=0
Jul 28 21:14:36.719183 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.718732    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"
Jul 28 21:14:36.719183 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.718733    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr" event={"ID":"0472bae2-b64e-457b-bc38-8da3168cb6e3","Type":"ContainerDied","Data":"15fc93412c2e717c24642b053100aa09dc97f1613daaa4d17e4e467e055158b0"}
Jul 28 21:14:36.719183 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.718797    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr" event={"ID":"0472bae2-b64e-457b-bc38-8da3168cb6e3","Type":"ContainerDied","Data":"1029111bebf303984fe6539aa72b522f0a11ab4d2b13f4a202fed385434dc95f"}
Jul 28 21:14:36.719183 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.718830    2537 scope.go:117] "RemoveContainer" containerID="15fc93412c2e717c24642b053100aa09dc97f1613daaa4d17e4e467e055158b0"
Jul 28 21:14:36.728749 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.728727    2537 scope.go:117] "RemoveContainer" containerID="15fc93412c2e717c24642b053100aa09dc97f1613daaa4d17e4e467e055158b0"
Jul 28 21:14:36.729038 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:14:36.729020    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15fc93412c2e717c24642b053100aa09dc97f1613daaa4d17e4e467e055158b0\": container with ID starting with 15fc93412c2e717c24642b053100aa09dc97f1613daaa4d17e4e467e055158b0 not found: ID does not exist" containerID="15fc93412c2e717c24642b053100aa09dc97f1613daaa4d17e4e467e055158b0"
Jul 28 21:14:36.729105 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.729046    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15fc93412c2e717c24642b053100aa09dc97f1613daaa4d17e4e467e055158b0"} err="failed to get container status \"15fc93412c2e717c24642b053100aa09dc97f1613daaa4d17e4e467e055158b0\": rpc error: code = NotFound desc = could not find container \"15fc93412c2e717c24642b053100aa09dc97f1613daaa4d17e4e467e055158b0\": container with ID starting with 15fc93412c2e717c24642b053100aa09dc97f1613daaa4d17e4e467e055158b0 not found: ID does not exist"
Jul 28 21:14:36.740368 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.740348    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"]
Jul 28 21:14:36.743416 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:36.743396    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f1163e-ff4c95f55-cd6gr"]
Jul 28 21:14:37.660110 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:37.660076    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:38.021656 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:38.021578    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0472bae2-b64e-457b-bc38-8da3168cb6e3" path="/var/lib/kubelet/pods/0472bae2-b64e-457b-bc38-8da3168cb6e3/volumes"
Jul 28 21:14:39.666924 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.666893    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"]
Jul 28 21:14:39.676201 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.676138    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0472bae2-b64e-457b-bc38-8da3168cb6e3" containerName="maas-api"
Jul 28 21:14:39.677058 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.676177    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="0472bae2-b64e-457b-bc38-8da3168cb6e3" containerName="maas-api"
Jul 28 21:14:39.677421 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.677393    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="0472bae2-b64e-457b-bc38-8da3168cb6e3" containerName="maas-api"
Jul 28 21:14:39.683030 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.683003    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"]
Jul 28 21:14:39.683130 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.683106    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"
Jul 28 21:14:39.685921 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.685901    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-3ba07b96-serving-cert\""
Jul 28 21:14:39.768999 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.768959    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7tjz\" (UniqueName: \"kubernetes.io/projected/f654b90e-79a8-415d-81e3-fd35f497d807-kube-api-access-m7tjz\") pod \"maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq\" (UID: \"f654b90e-79a8-415d-81e3-fd35f497d807\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"
Jul 28 21:14:39.768999 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.769001    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f654b90e-79a8-415d-81e3-fd35f497d807-maas-api-tls\") pod \"maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq\" (UID: \"f654b90e-79a8-415d-81e3-fd35f497d807\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"
Jul 28 21:14:39.769264 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.769020    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f654b90e-79a8-415d-81e3-fd35f497d807-openshift-service-ca\") pod \"maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq\" (UID: \"f654b90e-79a8-415d-81e3-fd35f497d807\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"
Jul 28 21:14:39.870282 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.870245    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f654b90e-79a8-415d-81e3-fd35f497d807-maas-api-tls\") pod \"maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq\" (UID: \"f654b90e-79a8-415d-81e3-fd35f497d807\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"
Jul 28 21:14:39.870471 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.870287    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f654b90e-79a8-415d-81e3-fd35f497d807-openshift-service-ca\") pod \"maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq\" (UID: \"f654b90e-79a8-415d-81e3-fd35f497d807\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"
Jul 28 21:14:39.870471 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.870433    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7tjz\" (UniqueName: \"kubernetes.io/projected/f654b90e-79a8-415d-81e3-fd35f497d807-kube-api-access-m7tjz\") pod \"maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq\" (UID: \"f654b90e-79a8-415d-81e3-fd35f497d807\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"
Jul 28 21:14:39.870981 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.870961    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f654b90e-79a8-415d-81e3-fd35f497d807-openshift-service-ca\") pod \"maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq\" (UID: \"f654b90e-79a8-415d-81e3-fd35f497d807\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"
Jul 28 21:14:39.872586 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.872567    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f654b90e-79a8-415d-81e3-fd35f497d807-maas-api-tls\") pod \"maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq\" (UID: \"f654b90e-79a8-415d-81e3-fd35f497d807\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"
Jul 28 21:14:39.878600 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.878572    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7tjz\" (UniqueName: \"kubernetes.io/projected/f654b90e-79a8-415d-81e3-fd35f497d807-kube-api-access-m7tjz\") pod \"maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq\" (UID: \"f654b90e-79a8-415d-81e3-fd35f497d807\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"
Jul 28 21:14:39.977983 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.977913    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:39.993940 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:39.993919    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"
Jul 28 21:14:40.076304 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:40.076261    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:40.152061 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:40.152033    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"]
Jul 28 21:14:40.154665 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:40.154629    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:40.155571 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:14:40.155548    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf654b90e_79a8_415d_81e3_fd35f497d807.slice/crio-c6b6cfa72010c2c036832f1899755b8ebb9da34d0d7d1d42b0814fc7e5510095 WatchSource:0}: Error finding container c6b6cfa72010c2c036832f1899755b8ebb9da34d0d7d1d42b0814fc7e5510095: Status 404 returned error can't find the container with id c6b6cfa72010c2c036832f1899755b8ebb9da34d0d7d1d42b0814fc7e5510095
Jul 28 21:14:40.304209 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:40.304119    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:14:40.741118 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:40.741085    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq" event={"ID":"f654b90e-79a8-415d-81e3-fd35f497d807","Type":"ContainerStarted","Data":"fba3839f8d294da9f47d9b928aeaf59e4d32a3998bb3bcdef3c12b8c5566cb7b"}
Jul 28 21:14:40.741495 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:40.741136    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq" event={"ID":"f654b90e-79a8-415d-81e3-fd35f497d807","Type":"ContainerStarted","Data":"c6b6cfa72010c2c036832f1899755b8ebb9da34d0d7d1d42b0814fc7e5510095"}
Jul 28 21:14:40.741495 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:40.741172    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"
Jul 28 21:14:40.757542 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:40.757489    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq" podStartSLOduration=1.374999363 podStartE2EDuration="1.757472589s" totalImagesPullingTime="382.473226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:14:40.757095384 +0000 UTC m=+1641.374677290" watchObservedRunningTime="2026-07-28 21:14:40.757472589 +0000 UTC m=+1641.375054491"
Jul 28 21:14:41.206174 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.206149    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:41.280281 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.280211    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a94bf216-c711-488c-b443-e288a7c0be87-istio-podinfo\") pod \"a94bf216-c711-488c-b443-e288a7c0be87\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") "
Jul 28 21:14:41.280281 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.280259    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a94bf216-c711-488c-b443-e288a7c0be87-istiod-ca-cert\") pod \"a94bf216-c711-488c-b443-e288a7c0be87\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") "
Jul 28 21:14:41.280500 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.280289    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-istio-data\") pod \"a94bf216-c711-488c-b443-e288a7c0be87\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") "
Jul 28 21:14:41.280500 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.280311    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-credential-socket\") pod \"a94bf216-c711-488c-b443-e288a7c0be87\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") "
Jul 28 21:14:41.280500 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.280362    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9b8ff\" (UniqueName: \"kubernetes.io/projected/a94bf216-c711-488c-b443-e288a7c0be87-kube-api-access-9b8ff\") pod \"a94bf216-c711-488c-b443-e288a7c0be87\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") "
Jul 28 21:14:41.280500 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.280387    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-workload-certs\") pod \"a94bf216-c711-488c-b443-e288a7c0be87\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") "
Jul 28 21:14:41.280500 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.280416    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-istio-envoy\") pod \"a94bf216-c711-488c-b443-e288a7c0be87\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") "
Jul 28 21:14:41.280500 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.280439    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a94bf216-c711-488c-b443-e288a7c0be87-istio-token\") pod \"a94bf216-c711-488c-b443-e288a7c0be87\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") "
Jul 28 21:14:41.280500 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.280464    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-workload-socket\") pod \"a94bf216-c711-488c-b443-e288a7c0be87\" (UID: \"a94bf216-c711-488c-b443-e288a7c0be87\") "
Jul 28 21:14:41.280841 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.280644    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94bf216-c711-488c-b443-e288a7c0be87-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a94bf216-c711-488c-b443-e288a7c0be87" (UID: "a94bf216-c711-488c-b443-e288a7c0be87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:14:41.280841 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.280658    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a94bf216-c711-488c-b443-e288a7c0be87" (UID: "a94bf216-c711-488c-b443-e288a7c0be87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:14:41.280841 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.280777    2537 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a94bf216-c711-488c-b443-e288a7c0be87-istiod-ca-cert\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:14:41.280841 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.280829    2537 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-credential-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:14:41.281095 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.280867    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a94bf216-c711-488c-b443-e288a7c0be87" (UID: "a94bf216-c711-488c-b443-e288a7c0be87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:14:41.281095 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.280971    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a94bf216-c711-488c-b443-e288a7c0be87" (UID: "a94bf216-c711-488c-b443-e288a7c0be87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:14:41.281211 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.281183    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-istio-data" (OuterVolumeSpecName: "istio-data") pod "a94bf216-c711-488c-b443-e288a7c0be87" (UID: "a94bf216-c711-488c-b443-e288a7c0be87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:14:41.282710 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.282681    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94bf216-c711-488c-b443-e288a7c0be87-kube-api-access-9b8ff" (OuterVolumeSpecName: "kube-api-access-9b8ff") pod "a94bf216-c711-488c-b443-e288a7c0be87" (UID: "a94bf216-c711-488c-b443-e288a7c0be87"). InnerVolumeSpecName "kube-api-access-9b8ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:14:41.282832 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.282699    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94bf216-c711-488c-b443-e288a7c0be87-istio-token" (OuterVolumeSpecName: "istio-token") pod "a94bf216-c711-488c-b443-e288a7c0be87" (UID: "a94bf216-c711-488c-b443-e288a7c0be87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:14:41.282832 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.282789    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a94bf216-c711-488c-b443-e288a7c0be87" (UID: "a94bf216-c711-488c-b443-e288a7c0be87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:14:41.282832 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.282794    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a94bf216-c711-488c-b443-e288a7c0be87-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a94bf216-c711-488c-b443-e288a7c0be87" (UID: "a94bf216-c711-488c-b443-e288a7c0be87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 21:14:41.381432 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.381404    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b8ff\" (UniqueName: \"kubernetes.io/projected/a94bf216-c711-488c-b443-e288a7c0be87-kube-api-access-9b8ff\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:14:41.381432 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.381429    2537 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-workload-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:14:41.381432 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.381438    2537 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-istio-envoy\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:14:41.381661 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.381447    2537 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a94bf216-c711-488c-b443-e288a7c0be87-istio-token\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:14:41.381661 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.381456    2537 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-workload-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:14:41.381661 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.381463    2537 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a94bf216-c711-488c-b443-e288a7c0be87-istio-podinfo\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:14:41.381661 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.381471    2537 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a94bf216-c711-488c-b443-e288a7c0be87-istio-data\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:14:41.748134 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.748102    2537 generic.go:356] "Generic (PLEG): container finished" podID="a94bf216-c711-488c-b443-e288a7c0be87" containerID="d9312a85a7a957d8810c2e85ba310d474499786a20525e52e23e63c1f44afd7a" exitCode=0
Jul 28 21:14:41.748564 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.748156    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"
Jul 28 21:14:41.748564 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.748186    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg" event={"ID":"a94bf216-c711-488c-b443-e288a7c0be87","Type":"ContainerDied","Data":"d9312a85a7a957d8810c2e85ba310d474499786a20525e52e23e63c1f44afd7a"}
Jul 28 21:14:41.748564 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.748242    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg" event={"ID":"a94bf216-c711-488c-b443-e288a7c0be87","Type":"ContainerDied","Data":"e3e8116ef00a7335513831beada88becffce384a61c49cc9c47dbf0d6f1ed6a8"}
Jul 28 21:14:41.748564 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.748273    2537 scope.go:117] "RemoveContainer" containerID="d9312a85a7a957d8810c2e85ba310d474499786a20525e52e23e63c1f44afd7a"
Jul 28 21:14:41.757839 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.757823    2537 scope.go:117] "RemoveContainer" containerID="d3b33a1d975dd5fd9f3f7defb8267cbe266a7e720b878fbc98460216f678e292"
Jul 28 21:14:41.765760 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.765742    2537 scope.go:117] "RemoveContainer" containerID="d9312a85a7a957d8810c2e85ba310d474499786a20525e52e23e63c1f44afd7a"
Jul 28 21:14:41.765994 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:14:41.765975    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9312a85a7a957d8810c2e85ba310d474499786a20525e52e23e63c1f44afd7a\": container with ID starting with d9312a85a7a957d8810c2e85ba310d474499786a20525e52e23e63c1f44afd7a not found: ID does not exist" containerID="d9312a85a7a957d8810c2e85ba310d474499786a20525e52e23e63c1f44afd7a"
Jul 28 21:14:41.766082 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.766007    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9312a85a7a957d8810c2e85ba310d474499786a20525e52e23e63c1f44afd7a"} err="failed to get container status \"d9312a85a7a957d8810c2e85ba310d474499786a20525e52e23e63c1f44afd7a\": rpc error: code = NotFound desc = could not find container \"d9312a85a7a957d8810c2e85ba310d474499786a20525e52e23e63c1f44afd7a\": container with ID starting with d9312a85a7a957d8810c2e85ba310d474499786a20525e52e23e63c1f44afd7a not found: ID does not exist"
Jul 28 21:14:41.766082 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.766031    2537 scope.go:117] "RemoveContainer" containerID="d3b33a1d975dd5fd9f3f7defb8267cbe266a7e720b878fbc98460216f678e292"
Jul 28 21:14:41.766277 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:14:41.766262    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b33a1d975dd5fd9f3f7defb8267cbe266a7e720b878fbc98460216f678e292\": container with ID starting with d3b33a1d975dd5fd9f3f7defb8267cbe266a7e720b878fbc98460216f678e292 not found: ID does not exist" containerID="d3b33a1d975dd5fd9f3f7defb8267cbe266a7e720b878fbc98460216f678e292"
Jul 28 21:14:41.766328 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.766283    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b33a1d975dd5fd9f3f7defb8267cbe266a7e720b878fbc98460216f678e292"} err="failed to get container status \"d3b33a1d975dd5fd9f3f7defb8267cbe266a7e720b878fbc98460216f678e292\": rpc error: code = NotFound desc = could not find container \"d3b33a1d975dd5fd9f3f7defb8267cbe266a7e720b878fbc98460216f678e292\": container with ID starting with d3b33a1d975dd5fd9f3f7defb8267cbe266a7e720b878fbc98460216f678e292 not found: ID does not exist"
Jul 28 21:14:41.769600 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.769577    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"]
Jul 28 21:14:41.772883 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:41.772845    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a4f1163e-openshift-default-79cfd6fd9d-jz9cg"]
Jul 28 21:14:42.020955 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:42.020877    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a94bf216-c711-488c-b443-e288a7c0be87" path="/var/lib/kubelet/pods/a94bf216-c711-488c-b443-e288a7c0be87/volumes"
Jul 28 21:14:46.753319 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:14:46.753288    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"
Jul 28 21:15:10.868941 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:10.868891    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"]
Jul 28 21:15:10.869579 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:10.869143    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq" podUID="f654b90e-79a8-415d-81e3-fd35f497d807" containerName="maas-api" containerID="cri-o://fba3839f8d294da9f47d9b928aeaf59e4d32a3998bb3bcdef3c12b8c5566cb7b" gracePeriod=30
Jul 28 21:15:10.901836 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:10.901803    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:11.122320 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.122263    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"
Jul 28 21:15:11.164460 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.164414    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:11.229736 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.229695    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:11.238640 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.238606    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f654b90e-79a8-415d-81e3-fd35f497d807-openshift-service-ca\") pod \"f654b90e-79a8-415d-81e3-fd35f497d807\" (UID: \"f654b90e-79a8-415d-81e3-fd35f497d807\") "
Jul 28 21:15:11.238757 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.238650    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f654b90e-79a8-415d-81e3-fd35f497d807-maas-api-tls\") pod \"f654b90e-79a8-415d-81e3-fd35f497d807\" (UID: \"f654b90e-79a8-415d-81e3-fd35f497d807\") "
Jul 28 21:15:11.238757 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.238709    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7tjz\" (UniqueName: \"kubernetes.io/projected/f654b90e-79a8-415d-81e3-fd35f497d807-kube-api-access-m7tjz\") pod \"f654b90e-79a8-415d-81e3-fd35f497d807\" (UID: \"f654b90e-79a8-415d-81e3-fd35f497d807\") "
Jul 28 21:15:11.239114 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.239083    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f654b90e-79a8-415d-81e3-fd35f497d807-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f654b90e-79a8-415d-81e3-fd35f497d807" (UID: "f654b90e-79a8-415d-81e3-fd35f497d807"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:15:11.241361 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.241324    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f654b90e-79a8-415d-81e3-fd35f497d807-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f654b90e-79a8-415d-81e3-fd35f497d807" (UID: "f654b90e-79a8-415d-81e3-fd35f497d807"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:15:11.241601 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.241582    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f654b90e-79a8-415d-81e3-fd35f497d807-kube-api-access-m7tjz" (OuterVolumeSpecName: "kube-api-access-m7tjz") pod "f654b90e-79a8-415d-81e3-fd35f497d807" (UID: "f654b90e-79a8-415d-81e3-fd35f497d807"). InnerVolumeSpecName "kube-api-access-m7tjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:15:11.340211 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.340171    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f654b90e-79a8-415d-81e3-fd35f497d807-openshift-service-ca\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:15:11.340211 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.340203    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f654b90e-79a8-415d-81e3-fd35f497d807-maas-api-tls\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:15:11.340211 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.340216    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7tjz\" (UniqueName: \"kubernetes.io/projected/f654b90e-79a8-415d-81e3-fd35f497d807-kube-api-access-m7tjz\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:15:11.509202 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.509095    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:11.659159 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.659124    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:11.907919 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.907882    2537 generic.go:356] "Generic (PLEG): container finished" podID="f654b90e-79a8-415d-81e3-fd35f497d807" containerID="fba3839f8d294da9f47d9b928aeaf59e4d32a3998bb3bcdef3c12b8c5566cb7b" exitCode=0
Jul 28 21:15:11.908409 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.907924    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"
Jul 28 21:15:11.908409 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.907952    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq" event={"ID":"f654b90e-79a8-415d-81e3-fd35f497d807","Type":"ContainerDied","Data":"fba3839f8d294da9f47d9b928aeaf59e4d32a3998bb3bcdef3c12b8c5566cb7b"}
Jul 28 21:15:11.908409 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.908039    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq" event={"ID":"f654b90e-79a8-415d-81e3-fd35f497d807","Type":"ContainerDied","Data":"c6b6cfa72010c2c036832f1899755b8ebb9da34d0d7d1d42b0814fc7e5510095"}
Jul 28 21:15:11.908409 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.908068    2537 scope.go:117] "RemoveContainer" containerID="fba3839f8d294da9f47d9b928aeaf59e4d32a3998bb3bcdef3c12b8c5566cb7b"
Jul 28 21:15:11.918366 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.918345    2537 scope.go:117] "RemoveContainer" containerID="fba3839f8d294da9f47d9b928aeaf59e4d32a3998bb3bcdef3c12b8c5566cb7b"
Jul 28 21:15:11.918640 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:15:11.918607    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba3839f8d294da9f47d9b928aeaf59e4d32a3998bb3bcdef3c12b8c5566cb7b\": container with ID starting with fba3839f8d294da9f47d9b928aeaf59e4d32a3998bb3bcdef3c12b8c5566cb7b not found: ID does not exist" containerID="fba3839f8d294da9f47d9b928aeaf59e4d32a3998bb3bcdef3c12b8c5566cb7b"
Jul 28 21:15:11.918755 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.918651    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba3839f8d294da9f47d9b928aeaf59e4d32a3998bb3bcdef3c12b8c5566cb7b"} err="failed to get container status \"fba3839f8d294da9f47d9b928aeaf59e4d32a3998bb3bcdef3c12b8c5566cb7b\": rpc error: code = NotFound desc = could not find container \"fba3839f8d294da9f47d9b928aeaf59e4d32a3998bb3bcdef3c12b8c5566cb7b\": container with ID starting with fba3839f8d294da9f47d9b928aeaf59e4d32a3998bb3bcdef3c12b8c5566cb7b not found: ID does not exist"
Jul 28 21:15:11.935402 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.935377    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"]
Jul 28 21:15:11.935745 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.935733    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94bf216-c711-488c-b443-e288a7c0be87" containerName="istio-proxy"
Jul 28 21:15:11.935799 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.935748    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94bf216-c711-488c-b443-e288a7c0be87" containerName="istio-proxy"
Jul 28 21:15:11.935799 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.935762    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f654b90e-79a8-415d-81e3-fd35f497d807" containerName="maas-api"
Jul 28 21:15:11.935799 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.935768    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="f654b90e-79a8-415d-81e3-fd35f497d807" containerName="maas-api"
Jul 28 21:15:11.935799 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.935780    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94bf216-c711-488c-b443-e288a7c0be87" containerName="istio-proxy"
Jul 28 21:15:11.935799 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.935786    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94bf216-c711-488c-b443-e288a7c0be87" containerName="istio-proxy"
Jul 28 21:15:11.936001 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.935842    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94bf216-c711-488c-b443-e288a7c0be87" containerName="istio-proxy"
Jul 28 21:15:11.936001 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.935870    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="f654b90e-79a8-415d-81e3-fd35f497d807" containerName="maas-api"
Jul 28 21:15:11.940163 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.940143    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"]
Jul 28 21:15:11.940255 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.940223    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:11.942681 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.942663    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-37fe5a92-openshift-default-dockercfg-ll2dd\""
Jul 28 21:15:11.946374 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.945793    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3ba07b96-7c645bb7d5-vqhqq"]
Jul 28 21:15:11.948777 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.948566    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"]
Jul 28 21:15:11.956337 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:11.956316    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:12.022027 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.021996    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f654b90e-79a8-415d-81e3-fd35f497d807" path="/var/lib/kubelet/pods/f654b90e-79a8-415d-81e3-fd35f497d807/volumes"
Jul 28 21:15:12.047570 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.047532    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5fzc\" (UniqueName: \"kubernetes.io/projected/ac1d2948-b635-47b5-b94e-e74b6779083a-kube-api-access-t5fzc\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.047723 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.047584    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-podinfo\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.047723 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.047609    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1d2948-b635-47b5-b94e-e74b6779083a-istiod-ca-cert\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.047723 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.047676    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-workload-certs\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.047723 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.047717    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-credential-socket\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.047967 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.047739    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-workload-socket\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.047967 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.047756    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-data\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.047967 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.047790    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-token\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.047967 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.047889    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-envoy\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.148703 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.148668    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5fzc\" (UniqueName: \"kubernetes.io/projected/ac1d2948-b635-47b5-b94e-e74b6779083a-kube-api-access-t5fzc\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.148910 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.148721    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-podinfo\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.148910 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.148737    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1d2948-b635-47b5-b94e-e74b6779083a-istiod-ca-cert\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.148910 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.148832    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-workload-certs\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.149135 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.149113    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-credential-socket\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.149193 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.149160    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-workload-socket\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.149249 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.149189    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-data\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.149249 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.149214    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-token\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.149355 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.149278    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-envoy\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.149355 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.149289    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-workload-certs\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.149468 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.149388    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-credential-socket\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.149525 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.149487    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-workload-socket\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.149582 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.149565    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1d2948-b635-47b5-b94e-e74b6779083a-istiod-ca-cert\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.149729 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.149709    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-data\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.151356 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.151326    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-podinfo\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.151437 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.151388    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-envoy\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.156990 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.156971    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-token\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.157111 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.157094    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5fzc\" (UniqueName: \"kubernetes.io/projected/ac1d2948-b635-47b5-b94e-e74b6779083a-kube-api-access-t5fzc\") pod \"e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") " pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.252675 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.252617    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:12.388309 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.388287    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"]
Jul 28 21:15:12.390332 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:15:12.390293    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1d2948_b635_47b5_b94e_e74b6779083a.slice/crio-37803625ef5efd3be9ceb62d91880e74e8e44d3e22577a4ab12cd5808f1214e6 WatchSource:0}: Error finding container 37803625ef5efd3be9ceb62d91880e74e8e44d3e22577a4ab12cd5808f1214e6: Status 404 returned error can't find the container with id 37803625ef5efd3be9ceb62d91880e74e8e44d3e22577a4ab12cd5808f1214e6
Jul 28 21:15:12.392488 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.392454    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:15:12.392582 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.392519    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:15:12.392582 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.392547    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:15:12.474535 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.474507    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:12.916477 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.916445    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8" event={"ID":"ac1d2948-b635-47b5-b94e-e74b6779083a","Type":"ContainerStarted","Data":"0ceb1f93a0f412e11a2fae334687d7af286a8e794220d3b23a326fb455fdec83"}
Jul 28 21:15:12.916879 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.916494    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8" event={"ID":"ac1d2948-b635-47b5-b94e-e74b6779083a","Type":"ContainerStarted","Data":"37803625ef5efd3be9ceb62d91880e74e8e44d3e22577a4ab12cd5808f1214e6"}
Jul 28 21:15:12.938846 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:12.938798    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8" podStartSLOduration=1.9387829810000001 podStartE2EDuration="1.938782981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:15:12.937234795 +0000 UTC m=+1673.554816697" watchObservedRunningTime="2026-07-28 21:15:12.938782981 +0000 UTC m=+1673.556364882"
Jul 28 21:15:13.253710 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:13.253632    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:13.258206 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:13.258176    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:13.921092 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:13.921062    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:13.921982 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:13.921963    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:14.292351 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.292250    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:14.395287 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.395250    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:14.536595 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.536566    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"]
Jul 28 21:15:14.537053 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.537039    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94bf216-c711-488c-b443-e288a7c0be87" containerName="istio-proxy"
Jul 28 21:15:14.540191 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.540173    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"
Jul 28 21:15:14.543668 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.543614    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-37fe5a92-serving-cert\""
Jul 28 21:15:14.559388 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.559357    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"]
Jul 28 21:15:14.596964 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.596932    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:14.676481 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.676450    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-openshift-service-ca\") pod \"maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9\" (UID: \"90cf49ec-6604-46b6-bcf8-54d7b4312ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"
Jul 28 21:15:14.676641 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.676533    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwmb9\" (UniqueName: \"kubernetes.io/projected/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-kube-api-access-zwmb9\") pod \"maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9\" (UID: \"90cf49ec-6604-46b6-bcf8-54d7b4312ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"
Jul 28 21:15:14.676699 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.676629    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-maas-api-tls\") pod \"maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9\" (UID: \"90cf49ec-6604-46b6-bcf8-54d7b4312ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"
Jul 28 21:15:14.693336 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.693307    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:14.777417 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.777386    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-openshift-service-ca\") pod \"maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9\" (UID: \"90cf49ec-6604-46b6-bcf8-54d7b4312ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"
Jul 28 21:15:14.777545 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.777460    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwmb9\" (UniqueName: \"kubernetes.io/projected/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-kube-api-access-zwmb9\") pod \"maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9\" (UID: \"90cf49ec-6604-46b6-bcf8-54d7b4312ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"
Jul 28 21:15:14.777636 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.777586    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-maas-api-tls\") pod \"maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9\" (UID: \"90cf49ec-6604-46b6-bcf8-54d7b4312ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"
Jul 28 21:15:14.778041 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.778021    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-openshift-service-ca\") pod \"maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9\" (UID: \"90cf49ec-6604-46b6-bcf8-54d7b4312ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"
Jul 28 21:15:14.779937 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.779920    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-maas-api-tls\") pod \"maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9\" (UID: \"90cf49ec-6604-46b6-bcf8-54d7b4312ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"
Jul 28 21:15:14.787542 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.787513    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwmb9\" (UniqueName: \"kubernetes.io/projected/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-kube-api-access-zwmb9\") pod \"maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9\" (UID: \"90cf49ec-6604-46b6-bcf8-54d7b4312ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"
Jul 28 21:15:14.850333 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.850311    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"
Jul 28 21:15:14.985317 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:14.985294    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"]
Jul 28 21:15:14.987343 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:15:14.987308    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cf49ec_6604_46b6_bcf8_54d7b4312ea1.slice/crio-4e85a1fe296e62230c43e89199557da195d1c5d4bd45baeae83286528b924c4a WatchSource:0}: Error finding container 4e85a1fe296e62230c43e89199557da195d1c5d4bd45baeae83286528b924c4a: Status 404 returned error can't find the container with id 4e85a1fe296e62230c43e89199557da195d1c5d4bd45baeae83286528b924c4a
Jul 28 21:15:15.084365 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:15.084334    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:15.180636 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:15.180543    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:15.932433 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:15.932397    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9" event={"ID":"90cf49ec-6604-46b6-bcf8-54d7b4312ea1","Type":"ContainerStarted","Data":"82df5726ebb0ea3e92f117977085ca2c797c5c96386fec448b4c2650fa41a3c8"}
Jul 28 21:15:15.932607 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:15.932461    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9" event={"ID":"90cf49ec-6604-46b6-bcf8-54d7b4312ea1","Type":"ContainerStarted","Data":"4e85a1fe296e62230c43e89199557da195d1c5d4bd45baeae83286528b924c4a"}
Jul 28 21:15:15.932607 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:15.932488    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"
Jul 28 21:15:15.949903 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:15.949833    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9" podStartSLOduration=1.443319525 podStartE2EDuration="1.949814365s" totalImagesPullingTime="506.49484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:15:15.949651997 +0000 UTC m=+1676.567233900" watchObservedRunningTime="2026-07-28 21:15:15.949814365 +0000 UTC m=+1676.567396270"
Jul 28 21:15:21.943652 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:21.943623    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"
Jul 28 21:15:28.923126 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:28.923088    2537 patch_prober.go:28] interesting pod/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 28 21:15:28.923549 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:28.923152    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8" podUID="ac1d2948-b635-47b5-b94e-e74b6779083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 28 21:15:29.004087 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:29.004056    2537 generic.go:356] "Generic (PLEG): container finished" podID="ac1d2948-b635-47b5-b94e-e74b6779083a" containerID="0ceb1f93a0f412e11a2fae334687d7af286a8e794220d3b23a326fb455fdec83" exitCode=0
Jul 28 21:15:29.004225 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:29.004117    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8" event={"ID":"ac1d2948-b635-47b5-b94e-e74b6779083a","Type":"ContainerDied","Data":"0ceb1f93a0f412e11a2fae334687d7af286a8e794220d3b23a326fb455fdec83"}
Jul 28 21:15:29.004452 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:29.004438    2537 scope.go:117] "RemoveContainer" containerID="0ceb1f93a0f412e11a2fae334687d7af286a8e794220d3b23a326fb455fdec83"
Jul 28 21:15:29.005422 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:29.005392    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:15:29.005476 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:29.005465    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:15:29.005514 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:29.005498    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:15:30.012807 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:30.012751    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8" event={"ID":"ac1d2948-b635-47b5-b94e-e74b6779083a","Type":"ContainerStarted","Data":"81e394e7aaf88da53b6ac7790fb79b478bc1de01dd07c3f68c12857ab7f446ad"}
Jul 28 21:15:30.253363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:30.253326    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:30.253580 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:30.253557    2537 patch_prober.go:28] interesting pod/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 28 21:15:30.253627 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:30.253606    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8" podUID="ac1d2948-b635-47b5-b94e-e74b6779083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 28 21:15:31.253015 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:31.252982    2537 patch_prober.go:28] interesting pod/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 28 21:15:31.253410 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:31.253037    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8" podUID="ac1d2948-b635-47b5-b94e-e74b6779083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 28 21:15:32.253458 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:32.253417    2537 patch_prober.go:28] interesting pod/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 28 21:15:32.253892 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:32.253491    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8" podUID="ac1d2948-b635-47b5-b94e-e74b6779083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 28 21:15:33.252910 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:33.252873    2537 patch_prober.go:28] interesting pod/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 28 21:15:33.253089 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:33.252936    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8" podUID="ac1d2948-b635-47b5-b94e-e74b6779083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 28 21:15:34.258090 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:34.258056    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:34.258531 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:34.258310    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:34.258954 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:34.258937    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:45.164666 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.164582    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"]
Jul 28 21:15:45.165172 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.164831    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9" podUID="90cf49ec-6604-46b6-bcf8-54d7b4312ea1" containerName="maas-api" containerID="cri-o://82df5726ebb0ea3e92f117977085ca2c797c5c96386fec448b4c2650fa41a3c8" gracePeriod=30
Jul 28 21:15:45.186562 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.186534    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:45.262200 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.262162    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:45.414509 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.414482    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"
Jul 28 21:15:45.467794 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.467711    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:45.537910 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.537877    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwmb9\" (UniqueName: \"kubernetes.io/projected/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-kube-api-access-zwmb9\") pod \"90cf49ec-6604-46b6-bcf8-54d7b4312ea1\" (UID: \"90cf49ec-6604-46b6-bcf8-54d7b4312ea1\") "
Jul 28 21:15:45.538069 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.537930    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-openshift-service-ca\") pod \"90cf49ec-6604-46b6-bcf8-54d7b4312ea1\" (UID: \"90cf49ec-6604-46b6-bcf8-54d7b4312ea1\") "
Jul 28 21:15:45.538069 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.537950    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-maas-api-tls\") pod \"90cf49ec-6604-46b6-bcf8-54d7b4312ea1\" (UID: \"90cf49ec-6604-46b6-bcf8-54d7b4312ea1\") "
Jul 28 21:15:45.538317 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.538287    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90cf49ec-6604-46b6-bcf8-54d7b4312ea1" (UID: "90cf49ec-6604-46b6-bcf8-54d7b4312ea1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:15:45.540056 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.540025    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-kube-api-access-zwmb9" (OuterVolumeSpecName: "kube-api-access-zwmb9") pod "90cf49ec-6604-46b6-bcf8-54d7b4312ea1" (UID: "90cf49ec-6604-46b6-bcf8-54d7b4312ea1"). InnerVolumeSpecName "kube-api-access-zwmb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:15:45.540150 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.540120    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90cf49ec-6604-46b6-bcf8-54d7b4312ea1" (UID: "90cf49ec-6604-46b6-bcf8-54d7b4312ea1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:15:45.639252 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.639211    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwmb9\" (UniqueName: \"kubernetes.io/projected/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-kube-api-access-zwmb9\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:15:45.639252 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.639251    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-openshift-service-ca\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:15:45.639252 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.639261    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cf49ec-6604-46b6-bcf8-54d7b4312ea1-maas-api-tls\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:15:45.922277 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:45.922233    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:46.115603 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:46.115573    2537 generic.go:356] "Generic (PLEG): container finished" podID="90cf49ec-6604-46b6-bcf8-54d7b4312ea1" containerID="82df5726ebb0ea3e92f117977085ca2c797c5c96386fec448b4c2650fa41a3c8" exitCode=0
Jul 28 21:15:46.115761 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:46.115628    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"
Jul 28 21:15:46.115761 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:46.115629    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9" event={"ID":"90cf49ec-6604-46b6-bcf8-54d7b4312ea1","Type":"ContainerDied","Data":"82df5726ebb0ea3e92f117977085ca2c797c5c96386fec448b4c2650fa41a3c8"}
Jul 28 21:15:46.115899 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:46.115770    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9" event={"ID":"90cf49ec-6604-46b6-bcf8-54d7b4312ea1","Type":"ContainerDied","Data":"4e85a1fe296e62230c43e89199557da195d1c5d4bd45baeae83286528b924c4a"}
Jul 28 21:15:46.115899 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:46.115811    2537 scope.go:117] "RemoveContainer" containerID="82df5726ebb0ea3e92f117977085ca2c797c5c96386fec448b4c2650fa41a3c8"
Jul 28 21:15:46.124978 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:46.124961    2537 scope.go:117] "RemoveContainer" containerID="82df5726ebb0ea3e92f117977085ca2c797c5c96386fec448b4c2650fa41a3c8"
Jul 28 21:15:46.125231 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:15:46.125208    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82df5726ebb0ea3e92f117977085ca2c797c5c96386fec448b4c2650fa41a3c8\": container with ID starting with 82df5726ebb0ea3e92f117977085ca2c797c5c96386fec448b4c2650fa41a3c8 not found: ID does not exist" containerID="82df5726ebb0ea3e92f117977085ca2c797c5c96386fec448b4c2650fa41a3c8"
Jul 28 21:15:46.125284 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:46.125239    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82df5726ebb0ea3e92f117977085ca2c797c5c96386fec448b4c2650fa41a3c8"} err="failed to get container status \"82df5726ebb0ea3e92f117977085ca2c797c5c96386fec448b4c2650fa41a3c8\": rpc error: code = NotFound desc = could not find container \"82df5726ebb0ea3e92f117977085ca2c797c5c96386fec448b4c2650fa41a3c8\": container with ID starting with 82df5726ebb0ea3e92f117977085ca2c797c5c96386fec448b4c2650fa41a3c8 not found: ID does not exist"
Jul 28 21:15:46.133767 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:46.133746    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"]
Jul 28 21:15:46.141768 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:46.141746    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37fe5a92-5bc5669b54-9vdx9"]
Jul 28 21:15:46.489177 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:46.489147    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"]
Jul 28 21:15:46.489619 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:46.489347    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8" podUID="ac1d2948-b635-47b5-b94e-e74b6779083a" containerName="istio-proxy" containerID="cri-o://81e394e7aaf88da53b6ac7790fb79b478bc1de01dd07c3f68c12857ab7f446ad" gracePeriod=30
Jul 28 21:15:46.511503 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:46.511452    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:46.723376 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:46.723333    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:46.943288 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:46.943255    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:47.552974 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:47.552909    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:47.652999 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:47.652969    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:48.026339 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:48.026300    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90cf49ec-6604-46b6-bcf8-54d7b4312ea1" path="/var/lib/kubelet/pods/90cf49ec-6604-46b6-bcf8-54d7b4312ea1/volumes"
Jul 28 21:15:49.261221 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:49.261175    2537 patch_prober.go:28] interesting pod/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 21:15:49.261613 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:49.261254    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8" podUID="ac1d2948-b635-47b5-b94e-e74b6779083a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 21:15:49.684703 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:49.684642    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:49.783586 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:49.783549    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:49.999487 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:49.999398    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"]
Jul 28 21:15:49.999911 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:49.999892    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90cf49ec-6604-46b6-bcf8-54d7b4312ea1" containerName="maas-api"
Jul 28 21:15:50.000014 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:49.999912    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="90cf49ec-6604-46b6-bcf8-54d7b4312ea1" containerName="maas-api"
Jul 28 21:15:50.000071 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.000028    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="90cf49ec-6604-46b6-bcf8-54d7b4312ea1" containerName="maas-api"
Jul 28 21:15:50.004388 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.004366    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:15:50.007139 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.007114    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-99e39a7a-serving-cert\""
Jul 28 21:15:50.022532 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.022508    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"]
Jul 28 21:15:50.035285 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.035254    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:50.077407 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.077375    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5092f434-1843-49d5-bb63-b34b609e1da7-maas-api-tls\") pod \"maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk\" (UID: \"5092f434-1843-49d5-bb63-b34b609e1da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:15:50.077407 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.077414    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcxhf\" (UniqueName: \"kubernetes.io/projected/5092f434-1843-49d5-bb63-b34b609e1da7-kube-api-access-bcxhf\") pod \"maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk\" (UID: \"5092f434-1843-49d5-bb63-b34b609e1da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:15:50.077644 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.077614    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5092f434-1843-49d5-bb63-b34b609e1da7-openshift-service-ca\") pod \"maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk\" (UID: \"5092f434-1843-49d5-bb63-b34b609e1da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:15:50.179140 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.179112    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5092f434-1843-49d5-bb63-b34b609e1da7-openshift-service-ca\") pod \"maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk\" (UID: \"5092f434-1843-49d5-bb63-b34b609e1da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:15:50.179306 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.179248    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5092f434-1843-49d5-bb63-b34b609e1da7-maas-api-tls\") pod \"maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk\" (UID: \"5092f434-1843-49d5-bb63-b34b609e1da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:15:50.179306 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.179268    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcxhf\" (UniqueName: \"kubernetes.io/projected/5092f434-1843-49d5-bb63-b34b609e1da7-kube-api-access-bcxhf\") pod \"maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk\" (UID: \"5092f434-1843-49d5-bb63-b34b609e1da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:15:50.179430 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:15:50.179409    2537 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-serving-cert: secret "maas-api-e2e-derive-99e39a7a-serving-cert" not found
Jul 28 21:15:50.179512 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:15:50.179500    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5092f434-1843-49d5-bb63-b34b609e1da7-maas-api-tls podName:5092f434-1843-49d5-bb63-b34b609e1da7 nodeName:}" failed. No retries permitted until 2026-07-28 21:15:50.679476409 +0000 UTC m=+1711.297058293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5092f434-1843-49d5-bb63-b34b609e1da7-maas-api-tls") pod "maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk" (UID: "5092f434-1843-49d5-bb63-b34b609e1da7") : secret "maas-api-e2e-derive-99e39a7a-serving-cert" not found
Jul 28 21:15:50.179744 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.179726    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5092f434-1843-49d5-bb63-b34b609e1da7-openshift-service-ca\") pod \"maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk\" (UID: \"5092f434-1843-49d5-bb63-b34b609e1da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:15:50.188388 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.188368    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcxhf\" (UniqueName: \"kubernetes.io/projected/5092f434-1843-49d5-bb63-b34b609e1da7-kube-api-access-bcxhf\") pod \"maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk\" (UID: \"5092f434-1843-49d5-bb63-b34b609e1da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:15:50.316600 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.316565    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:50.533373 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.533331    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:50.685060 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.684976    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5092f434-1843-49d5-bb63-b34b609e1da7-maas-api-tls\") pod \"maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk\" (UID: \"5092f434-1843-49d5-bb63-b34b609e1da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:15:50.687350 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.687330    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5092f434-1843-49d5-bb63-b34b609e1da7-maas-api-tls\") pod \"maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk\" (UID: \"5092f434-1843-49d5-bb63-b34b609e1da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:15:50.846425 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.846387    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:15:50.915415 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:50.915385    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:15:51.051582 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.051557    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"]
Jul 28 21:15:51.053632 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:15:51.053605    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5092f434_1843_49d5_bb63_b34b609e1da7.slice/crio-0e5e0900b312b59609cb5163f60b187ed73d71b20c88837086fbe6a7a4366a8b WatchSource:0}: Error finding container 0e5e0900b312b59609cb5163f60b187ed73d71b20c88837086fbe6a7a4366a8b: Status 404 returned error can't find the container with id 0e5e0900b312b59609cb5163f60b187ed73d71b20c88837086fbe6a7a4366a8b
Jul 28 21:15:51.144133 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.144104    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk" event={"ID":"5092f434-1843-49d5-bb63-b34b609e1da7","Type":"ContainerStarted","Data":"0e5e0900b312b59609cb5163f60b187ed73d71b20c88837086fbe6a7a4366a8b"}
Jul 28 21:15:51.746725 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.746700    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:51.794265 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.794229    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1d2948-b635-47b5-b94e-e74b6779083a-istiod-ca-cert\") pod \"ac1d2948-b635-47b5-b94e-e74b6779083a\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") "
Jul 28 21:15:51.794472 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.794278    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5fzc\" (UniqueName: \"kubernetes.io/projected/ac1d2948-b635-47b5-b94e-e74b6779083a-kube-api-access-t5fzc\") pod \"ac1d2948-b635-47b5-b94e-e74b6779083a\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") "
Jul 28 21:15:51.794472 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.794297    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-data\") pod \"ac1d2948-b635-47b5-b94e-e74b6779083a\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") "
Jul 28 21:15:51.794472 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.794322    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-credential-socket\") pod \"ac1d2948-b635-47b5-b94e-e74b6779083a\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") "
Jul 28 21:15:51.794687 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.794514    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-envoy\") pod \"ac1d2948-b635-47b5-b94e-e74b6779083a\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") "
Jul 28 21:15:51.794687 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.794551    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-workload-socket\") pod \"ac1d2948-b635-47b5-b94e-e74b6779083a\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") "
Jul 28 21:15:51.794687 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.794552    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ac1d2948-b635-47b5-b94e-e74b6779083a" (UID: "ac1d2948-b635-47b5-b94e-e74b6779083a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:15:51.794687 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.794586    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-workload-certs\") pod \"ac1d2948-b635-47b5-b94e-e74b6779083a\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") "
Jul 28 21:15:51.794687 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.794659    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-token\") pod \"ac1d2948-b635-47b5-b94e-e74b6779083a\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") "
Jul 28 21:15:51.794995 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.794706    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-podinfo\") pod \"ac1d2948-b635-47b5-b94e-e74b6779083a\" (UID: \"ac1d2948-b635-47b5-b94e-e74b6779083a\") "
Jul 28 21:15:51.794995 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.794649    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1d2948-b635-47b5-b94e-e74b6779083a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ac1d2948-b635-47b5-b94e-e74b6779083a" (UID: "ac1d2948-b635-47b5-b94e-e74b6779083a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:15:51.794995 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.794959    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ac1d2948-b635-47b5-b94e-e74b6779083a" (UID: "ac1d2948-b635-47b5-b94e-e74b6779083a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:15:51.794995 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.794877    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ac1d2948-b635-47b5-b94e-e74b6779083a" (UID: "ac1d2948-b635-47b5-b94e-e74b6779083a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:15:51.795204 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.795070    2537 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-credential-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:15:51.795204 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.795087    2537 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-workload-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:15:51.795204 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.795100    2537 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-workload-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:15:51.795204 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.795113    2537 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1d2948-b635-47b5-b94e-e74b6779083a-istiod-ca-cert\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:15:51.795406 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.795266    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-data" (OuterVolumeSpecName: "istio-data") pod "ac1d2948-b635-47b5-b94e-e74b6779083a" (UID: "ac1d2948-b635-47b5-b94e-e74b6779083a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:15:51.796946 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.796909    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac1d2948-b635-47b5-b94e-e74b6779083a" (UID: "ac1d2948-b635-47b5-b94e-e74b6779083a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:15:51.797057 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.796965    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ac1d2948-b635-47b5-b94e-e74b6779083a" (UID: "ac1d2948-b635-47b5-b94e-e74b6779083a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 21:15:51.797255 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.797229    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1d2948-b635-47b5-b94e-e74b6779083a-kube-api-access-t5fzc" (OuterVolumeSpecName: "kube-api-access-t5fzc") pod "ac1d2948-b635-47b5-b94e-e74b6779083a" (UID: "ac1d2948-b635-47b5-b94e-e74b6779083a"). InnerVolumeSpecName "kube-api-access-t5fzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:15:51.797544 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.797523    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ac1d2948-b635-47b5-b94e-e74b6779083a" (UID: "ac1d2948-b635-47b5-b94e-e74b6779083a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:15:51.902477 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.897705    2537 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-token\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:15:51.902477 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.897743    2537 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-podinfo\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:15:51.902477 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.897758    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5fzc\" (UniqueName: \"kubernetes.io/projected/ac1d2948-b635-47b5-b94e-e74b6779083a-kube-api-access-t5fzc\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:15:51.902477 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.897780    2537 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-data\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:15:51.902477 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:51.897794    2537 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1d2948-b635-47b5-b94e-e74b6779083a-istio-envoy\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:15:52.150493 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:52.150454    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk" event={"ID":"5092f434-1843-49d5-bb63-b34b609e1da7","Type":"ContainerStarted","Data":"fdd158954e1f96d6f4bb88adb413e4fccafccb4e34d411095ee2bc0b2a7f7582"}
Jul 28 21:15:52.150678 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:52.150578    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:15:52.152272 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:52.152248    2537 generic.go:356] "Generic (PLEG): container finished" podID="ac1d2948-b635-47b5-b94e-e74b6779083a" containerID="81e394e7aaf88da53b6ac7790fb79b478bc1de01dd07c3f68c12857ab7f446ad" exitCode=0
Jul 28 21:15:52.152398 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:52.152297    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"
Jul 28 21:15:52.152398 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:52.152312    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8" event={"ID":"ac1d2948-b635-47b5-b94e-e74b6779083a","Type":"ContainerDied","Data":"81e394e7aaf88da53b6ac7790fb79b478bc1de01dd07c3f68c12857ab7f446ad"}
Jul 28 21:15:52.152398 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:52.152357    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8" event={"ID":"ac1d2948-b635-47b5-b94e-e74b6779083a","Type":"ContainerDied","Data":"37803625ef5efd3be9ceb62d91880e74e8e44d3e22577a4ab12cd5808f1214e6"}
Jul 28 21:15:52.152398 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:52.152387    2537 scope.go:117] "RemoveContainer" containerID="81e394e7aaf88da53b6ac7790fb79b478bc1de01dd07c3f68c12857ab7f446ad"
Jul 28 21:15:52.161379 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:52.161360    2537 scope.go:117] "RemoveContainer" containerID="0ceb1f93a0f412e11a2fae334687d7af286a8e794220d3b23a326fb455fdec83"
Jul 28 21:15:52.168498 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:52.168458    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk" podStartSLOduration=2.7770158 podStartE2EDuration="3.168445968s" totalImagesPullingTime="391.430168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:15:52.167316655 +0000 UTC m=+1712.784898558" watchObservedRunningTime="2026-07-28 21:15:52.168445968 +0000 UTC m=+1712.786027870"
Jul 28 21:15:52.169662 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:52.169645    2537 scope.go:117] "RemoveContainer" containerID="81e394e7aaf88da53b6ac7790fb79b478bc1de01dd07c3f68c12857ab7f446ad"
Jul 28 21:15:52.169927 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:15:52.169908    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e394e7aaf88da53b6ac7790fb79b478bc1de01dd07c3f68c12857ab7f446ad\": container with ID starting with 81e394e7aaf88da53b6ac7790fb79b478bc1de01dd07c3f68c12857ab7f446ad not found: ID does not exist" containerID="81e394e7aaf88da53b6ac7790fb79b478bc1de01dd07c3f68c12857ab7f446ad"
Jul 28 21:15:52.170029 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:52.169938    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e394e7aaf88da53b6ac7790fb79b478bc1de01dd07c3f68c12857ab7f446ad"} err="failed to get container status \"81e394e7aaf88da53b6ac7790fb79b478bc1de01dd07c3f68c12857ab7f446ad\": rpc error: code = NotFound desc = could not find container \"81e394e7aaf88da53b6ac7790fb79b478bc1de01dd07c3f68c12857ab7f446ad\": container with ID starting with 81e394e7aaf88da53b6ac7790fb79b478bc1de01dd07c3f68c12857ab7f446ad not found: ID does not exist"
Jul 28 21:15:52.170029 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:52.169975    2537 scope.go:117] "RemoveContainer" containerID="0ceb1f93a0f412e11a2fae334687d7af286a8e794220d3b23a326fb455fdec83"
Jul 28 21:15:52.170212 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:15:52.170190    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ceb1f93a0f412e11a2fae334687d7af286a8e794220d3b23a326fb455fdec83\": container with ID starting with 0ceb1f93a0f412e11a2fae334687d7af286a8e794220d3b23a326fb455fdec83 not found: ID does not exist" containerID="0ceb1f93a0f412e11a2fae334687d7af286a8e794220d3b23a326fb455fdec83"
Jul 28 21:15:52.170260 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:52.170217    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ceb1f93a0f412e11a2fae334687d7af286a8e794220d3b23a326fb455fdec83"} err="failed to get container status \"0ceb1f93a0f412e11a2fae334687d7af286a8e794220d3b23a326fb455fdec83\": rpc error: code = NotFound desc = could not find container \"0ceb1f93a0f412e11a2fae334687d7af286a8e794220d3b23a326fb455fdec83\": container with ID starting with 0ceb1f93a0f412e11a2fae334687d7af286a8e794220d3b23a326fb455fdec83 not found: ID does not exist"
Jul 28 21:15:52.181480 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:52.181456    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"]
Jul 28 21:15:52.187816 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:52.187796    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-37fe5a92-openshift-default-667c897bf6-vjff8"]
Jul 28 21:15:54.021914 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:54.021878    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac1d2948-b635-47b5-b94e-e74b6779083a" path="/var/lib/kubelet/pods/ac1d2948-b635-47b5-b94e-e74b6779083a/volumes"
Jul 28 21:15:58.163486 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:15:58.163461    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:16:20.464301 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.464214    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"]
Jul 28 21:16:20.465448 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.465053    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk" podUID="5092f434-1843-49d5-bb63-b34b609e1da7" containerName="maas-api" containerID="cri-o://fdd158954e1f96d6f4bb88adb413e4fccafccb4e34d411095ee2bc0b2a7f7582" gracePeriod=30
Jul 28 21:16:20.485813 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.485782    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:16:20.573878 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.571795    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:16:20.634374 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.634296    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:16:20.691621 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.691588    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:16:20.725550 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.725493    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:16:20.824020 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.823974    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:16:20.840009 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.839979    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcxhf\" (UniqueName: \"kubernetes.io/projected/5092f434-1843-49d5-bb63-b34b609e1da7-kube-api-access-bcxhf\") pod \"5092f434-1843-49d5-bb63-b34b609e1da7\" (UID: \"5092f434-1843-49d5-bb63-b34b609e1da7\") "
Jul 28 21:16:20.840135 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.840024    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5092f434-1843-49d5-bb63-b34b609e1da7-openshift-service-ca\") pod \"5092f434-1843-49d5-bb63-b34b609e1da7\" (UID: \"5092f434-1843-49d5-bb63-b34b609e1da7\") "
Jul 28 21:16:20.840190 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.840136    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5092f434-1843-49d5-bb63-b34b609e1da7-maas-api-tls\") pod \"5092f434-1843-49d5-bb63-b34b609e1da7\" (UID: \"5092f434-1843-49d5-bb63-b34b609e1da7\") "
Jul 28 21:16:20.840422 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.840383    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5092f434-1843-49d5-bb63-b34b609e1da7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5092f434-1843-49d5-bb63-b34b609e1da7" (UID: "5092f434-1843-49d5-bb63-b34b609e1da7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:16:20.842150 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.842119    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5092f434-1843-49d5-bb63-b34b609e1da7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5092f434-1843-49d5-bb63-b34b609e1da7" (UID: "5092f434-1843-49d5-bb63-b34b609e1da7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:16:20.842249 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.842228    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5092f434-1843-49d5-bb63-b34b609e1da7-kube-api-access-bcxhf" (OuterVolumeSpecName: "kube-api-access-bcxhf") pod "5092f434-1843-49d5-bb63-b34b609e1da7" (UID: "5092f434-1843-49d5-bb63-b34b609e1da7"). InnerVolumeSpecName "kube-api-access-bcxhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:16:20.941654 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.941623    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5092f434-1843-49d5-bb63-b34b609e1da7-maas-api-tls\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:16:20.941654 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.941652    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcxhf\" (UniqueName: \"kubernetes.io/projected/5092f434-1843-49d5-bb63-b34b609e1da7-kube-api-access-bcxhf\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:16:20.941837 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:20.941661    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5092f434-1843-49d5-bb63-b34b609e1da7-openshift-service-ca\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:16:21.045555 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:21.045481    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:16:21.301500 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:21.301400    2537 generic.go:356] "Generic (PLEG): container finished" podID="5092f434-1843-49d5-bb63-b34b609e1da7" containerID="fdd158954e1f96d6f4bb88adb413e4fccafccb4e34d411095ee2bc0b2a7f7582" exitCode=0
Jul 28 21:16:21.301500 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:21.301454    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"
Jul 28 21:16:21.301698 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:21.301492    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk" event={"ID":"5092f434-1843-49d5-bb63-b34b609e1da7","Type":"ContainerDied","Data":"fdd158954e1f96d6f4bb88adb413e4fccafccb4e34d411095ee2bc0b2a7f7582"}
Jul 28 21:16:21.301698 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:21.301550    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk" event={"ID":"5092f434-1843-49d5-bb63-b34b609e1da7","Type":"ContainerDied","Data":"0e5e0900b312b59609cb5163f60b187ed73d71b20c88837086fbe6a7a4366a8b"}
Jul 28 21:16:21.301698 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:21.301578    2537 scope.go:117] "RemoveContainer" containerID="fdd158954e1f96d6f4bb88adb413e4fccafccb4e34d411095ee2bc0b2a7f7582"
Jul 28 21:16:21.312111 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:21.312093    2537 scope.go:117] "RemoveContainer" containerID="fdd158954e1f96d6f4bb88adb413e4fccafccb4e34d411095ee2bc0b2a7f7582"
Jul 28 21:16:21.312362 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:16:21.312345    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd158954e1f96d6f4bb88adb413e4fccafccb4e34d411095ee2bc0b2a7f7582\": container with ID starting with fdd158954e1f96d6f4bb88adb413e4fccafccb4e34d411095ee2bc0b2a7f7582 not found: ID does not exist" containerID="fdd158954e1f96d6f4bb88adb413e4fccafccb4e34d411095ee2bc0b2a7f7582"
Jul 28 21:16:21.312415 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:21.312369    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd158954e1f96d6f4bb88adb413e4fccafccb4e34d411095ee2bc0b2a7f7582"} err="failed to get container status \"fdd158954e1f96d6f4bb88adb413e4fccafccb4e34d411095ee2bc0b2a7f7582\": rpc error: code = NotFound desc = could not find container \"fdd158954e1f96d6f4bb88adb413e4fccafccb4e34d411095ee2bc0b2a7f7582\": container with ID starting with fdd158954e1f96d6f4bb88adb413e4fccafccb4e34d411095ee2bc0b2a7f7582 not found: ID does not exist"
Jul 28 21:16:21.333731 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:21.333706    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"]
Jul 28 21:16:21.339747 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:21.339725    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99e39a7a-6f5f557449-97pxk"]
Jul 28 21:16:21.586317 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:21.586288    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:16:21.690285 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:21.690237    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:16:22.022201 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:22.022122    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5092f434-1843-49d5-bb63-b34b609e1da7" path="/var/lib/kubelet/pods/5092f434-1843-49d5-bb63-b34b609e1da7/volumes"
Jul 28 21:16:23.098131 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:23.098095    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:16:29.971715 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:29.971681    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:16:30.548158 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:30.548126    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:16:30.649116 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:30.649060    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:16:30.750916 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:30.750879    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:16:33.620215 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:33.620178    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:16:56.685690 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:16:56.685650    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:03.944188 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:03.944144    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:03.999151 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:03.999114    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:04.073555 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:04.073521    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:04.172546 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:04.172516    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:19.974668 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:19.974637    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:17:19.977343 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:19.977321    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:17:20.012169 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:20.012135    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:26.672898 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:26.672847    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:26.744449 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:26.744410    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:27.024075 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:27.023984    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:27.139548 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:27.139514    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:37.458539 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:37.458505    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:38.254523 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:38.254486    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:44.471529 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:44.471488    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:45.169784 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:45.169745    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:45.283384 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:45.283343    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:45.374117 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:45.374078    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:45.472128 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:45.472031    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:56.970395 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:56.970343    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:57.065300 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:57.065242    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:57.165377 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:57.165328    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:17:58.885586 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:17:58.885552    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:06.016290 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:06.016256    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:06.116301 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:06.116257    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:06.685803 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:06.685762    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:08.129967 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:08.129920    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:12.972388 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:12.972343    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:13.072493 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.072451    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:13.171967 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.171929    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:13.468398 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.468362    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"]
Jul 28 21:18:13.469023 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.469000    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5092f434-1843-49d5-bb63-b34b609e1da7" containerName="maas-api"
Jul 28 21:18:13.469023 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.469024    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="5092f434-1843-49d5-bb63-b34b609e1da7" containerName="maas-api"
Jul 28 21:18:13.469206 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.469046    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac1d2948-b635-47b5-b94e-e74b6779083a" containerName="istio-proxy"
Jul 28 21:18:13.469206 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.469056    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac1d2948-b635-47b5-b94e-e74b6779083a" containerName="istio-proxy"
Jul 28 21:18:13.469206 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.469074    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac1d2948-b635-47b5-b94e-e74b6779083a" containerName="istio-proxy"
Jul 28 21:18:13.469206 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.469083    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac1d2948-b635-47b5-b94e-e74b6779083a" containerName="istio-proxy"
Jul 28 21:18:13.469206 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.469190    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac1d2948-b635-47b5-b94e-e74b6779083a" containerName="istio-proxy"
Jul 28 21:18:13.469206 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.469204    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="5092f434-1843-49d5-bb63-b34b609e1da7" containerName="maas-api"
Jul 28 21:18:13.472258 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.472237    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.475127 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.475107    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7c69f006-openshift-default-dockercfg-pkwpt\""
Jul 28 21:18:13.489994 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.489969    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"]
Jul 28 21:18:13.504674 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.504644    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:13.613696 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.613658    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6bpl\" (UniqueName: \"kubernetes.io/projected/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-kube-api-access-x6bpl\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.613882 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.613733    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-workload-certs\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.613882 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.613774    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-podinfo\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.613970 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.613899    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-envoy\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.613970 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.613949    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istiod-ca-cert\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.614041 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.614014    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-token\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.614076 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.614048    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-credential-socket\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.614112 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.614078    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-workload-socket\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.614146 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.614124    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-data\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.624488 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.624455    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:13.714988 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.714949    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-token\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.714988 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.714992    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-credential-socket\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.715258 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.715011    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-workload-socket\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.715258 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.715133    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-data\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.715258 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.715191    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6bpl\" (UniqueName: \"kubernetes.io/projected/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-kube-api-access-x6bpl\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.715425 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.715277    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-workload-certs\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.715683 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.715365    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-podinfo\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.715766 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.715418    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-credential-socket\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.715828 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.715447    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-workload-socket\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.715828 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.715478    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-data\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.715828 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.715634    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-workload-certs\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.715828 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.715747    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-envoy\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.716057 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.715849    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istiod-ca-cert\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.716417 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.716397    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istiod-ca-cert\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.717758 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.717731    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-envoy\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.717919 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.717903    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-podinfo\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.726563 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.726503    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-token\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.726730 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.726712    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6bpl\" (UniqueName: \"kubernetes.io/projected/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-kube-api-access-x6bpl\") pod \"e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") " pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.784733 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.784698    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:13.953958 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.953932    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"]
Jul 28 21:18:13.956220 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:18:13.956186    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fcbcf8_c27c_41b8_9218_73d075d19b2c.slice/crio-33b7e7357d8878aa57dbfc7e6b43738c9b0249c5808ab4bb32a6d2f0cdec71bb WatchSource:0}: Error finding container 33b7e7357d8878aa57dbfc7e6b43738c9b0249c5808ab4bb32a6d2f0cdec71bb: Status 404 returned error can't find the container with id 33b7e7357d8878aa57dbfc7e6b43738c9b0249c5808ab4bb32a6d2f0cdec71bb
Jul 28 21:18:13.958818 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.958774    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:18:13.958950 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.958906    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:18:13.959022 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:13.958955    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:18:14.868243 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:14.868204    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg" event={"ID":"d5fcbcf8-c27c-41b8-9218-73d075d19b2c","Type":"ContainerStarted","Data":"410b99082fd1d19da65fa57288e4930efcc6c8336091cb1eeefd2de012605b2a"}
Jul 28 21:18:14.868643 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:14.868269    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg" event={"ID":"d5fcbcf8-c27c-41b8-9218-73d075d19b2c","Type":"ContainerStarted","Data":"33b7e7357d8878aa57dbfc7e6b43738c9b0249c5808ab4bb32a6d2f0cdec71bb"}
Jul 28 21:18:14.890810 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:14.890760    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg" podStartSLOduration=1.8907443819999998 podStartE2EDuration="1.890744382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:18:14.888880421 +0000 UTC m=+1855.506462324" watchObservedRunningTime="2026-07-28 21:18:14.890744382 +0000 UTC m=+1855.508326284"
Jul 28 21:18:15.785555 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:15.785518    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:15.790495 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:15.790469    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:15.872618 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:15.872589    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:15.873552 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:15.873531    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:16.817237 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:16.817200    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:16.923705 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:16.923663    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:17.034395 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.034369    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"]
Jul 28 21:18:17.034834 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.034821    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac1d2948-b635-47b5-b94e-e74b6779083a" containerName="istio-proxy"
Jul 28 21:18:17.038129 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.038108    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:18:17.041208 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.041187    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7c69f006-serving-cert\""
Jul 28 21:18:17.044807 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.044784    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f02541-8772-4fa8-962c-85f970465222-maas-api-tls\") pod \"maas-api-e2e-mt-7c69f006-5579cbc94d-p566r\" (UID: \"00f02541-8772-4fa8-962c-85f970465222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:18:17.044937 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.044824    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn2v8\" (UniqueName: \"kubernetes.io/projected/00f02541-8772-4fa8-962c-85f970465222-kube-api-access-dn2v8\") pod \"maas-api-e2e-mt-7c69f006-5579cbc94d-p566r\" (UID: \"00f02541-8772-4fa8-962c-85f970465222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:18:17.044937 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.044933    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f02541-8772-4fa8-962c-85f970465222-openshift-service-ca\") pod \"maas-api-e2e-mt-7c69f006-5579cbc94d-p566r\" (UID: \"00f02541-8772-4fa8-962c-85f970465222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:18:17.052566 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.052543    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"]
Jul 28 21:18:17.079457 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.079394    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:17.145848 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.145816    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f02541-8772-4fa8-962c-85f970465222-maas-api-tls\") pod \"maas-api-e2e-mt-7c69f006-5579cbc94d-p566r\" (UID: \"00f02541-8772-4fa8-962c-85f970465222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:18:17.146038 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.145887    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn2v8\" (UniqueName: \"kubernetes.io/projected/00f02541-8772-4fa8-962c-85f970465222-kube-api-access-dn2v8\") pod \"maas-api-e2e-mt-7c69f006-5579cbc94d-p566r\" (UID: \"00f02541-8772-4fa8-962c-85f970465222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:18:17.146038 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.145938    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f02541-8772-4fa8-962c-85f970465222-openshift-service-ca\") pod \"maas-api-e2e-mt-7c69f006-5579cbc94d-p566r\" (UID: \"00f02541-8772-4fa8-962c-85f970465222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:18:17.146038 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:18:17.145955    2537 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-serving-cert: secret "maas-api-e2e-mt-7c69f006-serving-cert" not found
Jul 28 21:18:17.146038 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:18:17.146027    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f02541-8772-4fa8-962c-85f970465222-maas-api-tls podName:00f02541-8772-4fa8-962c-85f970465222 nodeName:}" failed. No retries permitted until 2026-07-28 21:18:17.646010224 +0000 UTC m=+1858.263592104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00f02541-8772-4fa8-962c-85f970465222-maas-api-tls") pod "maas-api-e2e-mt-7c69f006-5579cbc94d-p566r" (UID: "00f02541-8772-4fa8-962c-85f970465222") : secret "maas-api-e2e-mt-7c69f006-serving-cert" not found
Jul 28 21:18:17.146597 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.146578    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f02541-8772-4fa8-962c-85f970465222-openshift-service-ca\") pod \"maas-api-e2e-mt-7c69f006-5579cbc94d-p566r\" (UID: \"00f02541-8772-4fa8-962c-85f970465222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:18:17.161548 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.161523    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn2v8\" (UniqueName: \"kubernetes.io/projected/00f02541-8772-4fa8-962c-85f970465222-kube-api-access-dn2v8\") pod \"maas-api-e2e-mt-7c69f006-5579cbc94d-p566r\" (UID: \"00f02541-8772-4fa8-962c-85f970465222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:18:17.438844 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.438727    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:17.616219 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.616179    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:17.648994 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.648962    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f02541-8772-4fa8-962c-85f970465222-maas-api-tls\") pod \"maas-api-e2e-mt-7c69f006-5579cbc94d-p566r\" (UID: \"00f02541-8772-4fa8-962c-85f970465222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:18:17.651381 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.651361    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f02541-8772-4fa8-962c-85f970465222-maas-api-tls\") pod \"maas-api-e2e-mt-7c69f006-5579cbc94d-p566r\" (UID: \"00f02541-8772-4fa8-962c-85f970465222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:18:17.721042 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.720969    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:17.949143 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:17.949112    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:18:18.087850 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:18.087819    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"]
Jul 28 21:18:18.090589 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:18:18.090564    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f02541_8772_4fa8_962c_85f970465222.slice/crio-babd5a9cf6479ff4c53ed577e6998ef62880ea178596ccda43e602c22a533fa2 WatchSource:0}: Error finding container babd5a9cf6479ff4c53ed577e6998ef62880ea178596ccda43e602c22a533fa2: Status 404 returned error can't find the container with id babd5a9cf6479ff4c53ed577e6998ef62880ea178596ccda43e602c22a533fa2
Jul 28 21:18:18.893817 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:18.893780    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r" event={"ID":"00f02541-8772-4fa8-962c-85f970465222","Type":"ContainerStarted","Data":"026840300dded4a70a9dfd168cab4b62ec486102bfb0f7f4b9f902837f753e75"}
Jul 28 21:18:18.894006 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:18.893834    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r" event={"ID":"00f02541-8772-4fa8-962c-85f970465222","Type":"ContainerStarted","Data":"babd5a9cf6479ff4c53ed577e6998ef62880ea178596ccda43e602c22a533fa2"}
Jul 28 21:18:18.894006 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:18.893897    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:18:18.910973 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:18.910902    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r" podStartSLOduration=1.491000043 podStartE2EDuration="1.910881249s" totalImagesPullingTime="419.881206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:18:18.909723985 +0000 UTC m=+1859.527305888" watchObservedRunningTime="2026-07-28 21:18:18.910881249 +0000 UTC m=+1859.528463153"
Jul 28 21:18:24.903809 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:24.903780    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:18:31.310321 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:31.309412    2537 patch_prober.go:28] interesting pod/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": read tcp 10.133.0.2:58064->10.133.0.59:15021: read: connection reset by peer" start-of-body=
Jul 28 21:18:31.310321 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:31.309487    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg" podUID="d5fcbcf8-c27c-41b8-9218-73d075d19b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": read tcp 10.133.0.2:58064->10.133.0.59:15021: read: connection reset by peer"
Jul 28 21:18:31.970373 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:31.970338    2537 generic.go:356] "Generic (PLEG): container finished" podID="d5fcbcf8-c27c-41b8-9218-73d075d19b2c" containerID="410b99082fd1d19da65fa57288e4930efcc6c8336091cb1eeefd2de012605b2a" exitCode=0
Jul 28 21:18:31.970547 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:31.970407    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg" event={"ID":"d5fcbcf8-c27c-41b8-9218-73d075d19b2c","Type":"ContainerDied","Data":"410b99082fd1d19da65fa57288e4930efcc6c8336091cb1eeefd2de012605b2a"}
Jul 28 21:18:31.970760 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:31.970747    2537 scope.go:117] "RemoveContainer" containerID="410b99082fd1d19da65fa57288e4930efcc6c8336091cb1eeefd2de012605b2a"
Jul 28 21:18:31.971728 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:31.971696    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:18:31.971846 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:31.971765    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:18:31.971846 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:31.971798    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:18:32.977289 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:32.977257    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg" event={"ID":"d5fcbcf8-c27c-41b8-9218-73d075d19b2c","Type":"ContainerStarted","Data":"68d90209495970d00e6938ba2955f76fca56975f9614a1ba7b4710d699681f0f"}
Jul 28 21:18:33.785592 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:33.785553    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:33.785794 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:33.785770    2537 patch_prober.go:28] interesting pod/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 28 21:18:33.785879 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:33.785819    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg" podUID="d5fcbcf8-c27c-41b8-9218-73d075d19b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 28 21:18:34.785548 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:34.785507    2537 patch_prober.go:28] interesting pod/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 28 21:18:34.785981 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:34.785571    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg" podUID="d5fcbcf8-c27c-41b8-9218-73d075d19b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 28 21:18:35.785382 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:35.785344    2537 patch_prober.go:28] interesting pod/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 28 21:18:35.785562 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:35.785401    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg" podUID="d5fcbcf8-c27c-41b8-9218-73d075d19b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 28 21:18:36.789062 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:36.789026    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:36.789589 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:36.789414    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:36.789966 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:36.789946    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:18:41.738353 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:41.738313    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:42.372474 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:42.372437    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:42.474805 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:42.474767    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:42.781816 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:42.781699    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:48.852379 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:48.852338    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:48.913208 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:48.913172    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:59.646231 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:59.646200    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"]
Jul 28 21:18:59.646718 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:59.646414    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r" podUID="00f02541-8772-4fa8-962c-85f970465222" containerName="maas-api" containerID="cri-o://026840300dded4a70a9dfd168cab4b62ec486102bfb0f7f4b9f902837f753e75" gracePeriod=30
Jul 28 21:18:59.676985 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:59.676953    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:59.777810 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:59.777772    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:59.864440 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:59.864381    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:18:59.909530 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:59.909464    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:18:59.942824 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:59.942799    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn2v8\" (UniqueName: \"kubernetes.io/projected/00f02541-8772-4fa8-962c-85f970465222-kube-api-access-dn2v8\") pod \"00f02541-8772-4fa8-962c-85f970465222\" (UID: \"00f02541-8772-4fa8-962c-85f970465222\") "
Jul 28 21:18:59.942987 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:59.942834    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f02541-8772-4fa8-962c-85f970465222-maas-api-tls\") pod \"00f02541-8772-4fa8-962c-85f970465222\" (UID: \"00f02541-8772-4fa8-962c-85f970465222\") "
Jul 28 21:18:59.942987 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:59.942866    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f02541-8772-4fa8-962c-85f970465222-openshift-service-ca\") pod \"00f02541-8772-4fa8-962c-85f970465222\" (UID: \"00f02541-8772-4fa8-962c-85f970465222\") "
Jul 28 21:18:59.943238 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:59.943205    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f02541-8772-4fa8-962c-85f970465222-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "00f02541-8772-4fa8-962c-85f970465222" (UID: "00f02541-8772-4fa8-962c-85f970465222"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:18:59.944961 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:59.944929    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f02541-8772-4fa8-962c-85f970465222-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "00f02541-8772-4fa8-962c-85f970465222" (UID: "00f02541-8772-4fa8-962c-85f970465222"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:18:59.945058 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:18:59.944975    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f02541-8772-4fa8-962c-85f970465222-kube-api-access-dn2v8" (OuterVolumeSpecName: "kube-api-access-dn2v8") pod "00f02541-8772-4fa8-962c-85f970465222" (UID: "00f02541-8772-4fa8-962c-85f970465222"). InnerVolumeSpecName "kube-api-access-dn2v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:19:00.043830 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:00.043802    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn2v8\" (UniqueName: \"kubernetes.io/projected/00f02541-8772-4fa8-962c-85f970465222-kube-api-access-dn2v8\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:19:00.043830 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:00.043828    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f02541-8772-4fa8-962c-85f970465222-maas-api-tls\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:19:00.044021 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:00.043840    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f02541-8772-4fa8-962c-85f970465222-openshift-service-ca\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:19:00.113348 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:00.113311    2537 generic.go:356] "Generic (PLEG): container finished" podID="00f02541-8772-4fa8-962c-85f970465222" containerID="026840300dded4a70a9dfd168cab4b62ec486102bfb0f7f4b9f902837f753e75" exitCode=0
Jul 28 21:19:00.113514 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:00.113356    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"
Jul 28 21:19:00.113514 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:00.113399    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r" event={"ID":"00f02541-8772-4fa8-962c-85f970465222","Type":"ContainerDied","Data":"026840300dded4a70a9dfd168cab4b62ec486102bfb0f7f4b9f902837f753e75"}
Jul 28 21:19:00.113514 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:00.113457    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r" event={"ID":"00f02541-8772-4fa8-962c-85f970465222","Type":"ContainerDied","Data":"babd5a9cf6479ff4c53ed577e6998ef62880ea178596ccda43e602c22a533fa2"}
Jul 28 21:19:00.113514 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:00.113474    2537 scope.go:117] "RemoveContainer" containerID="026840300dded4a70a9dfd168cab4b62ec486102bfb0f7f4b9f902837f753e75"
Jul 28 21:19:00.122076 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:00.122060    2537 scope.go:117] "RemoveContainer" containerID="026840300dded4a70a9dfd168cab4b62ec486102bfb0f7f4b9f902837f753e75"
Jul 28 21:19:00.122344 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:19:00.122327    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026840300dded4a70a9dfd168cab4b62ec486102bfb0f7f4b9f902837f753e75\": container with ID starting with 026840300dded4a70a9dfd168cab4b62ec486102bfb0f7f4b9f902837f753e75 not found: ID does not exist" containerID="026840300dded4a70a9dfd168cab4b62ec486102bfb0f7f4b9f902837f753e75"
Jul 28 21:19:00.122389 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:00.122353    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026840300dded4a70a9dfd168cab4b62ec486102bfb0f7f4b9f902837f753e75"} err="failed to get container status \"026840300dded4a70a9dfd168cab4b62ec486102bfb0f7f4b9f902837f753e75\": rpc error: code = NotFound desc = could not find container \"026840300dded4a70a9dfd168cab4b62ec486102bfb0f7f4b9f902837f753e75\": container with ID starting with 026840300dded4a70a9dfd168cab4b62ec486102bfb0f7f4b9f902837f753e75 not found: ID does not exist"
Jul 28 21:19:00.132392 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:00.132369    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"]
Jul 28 21:19:00.136351 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:00.136330    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r"]
Jul 28 21:19:00.220558 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:00.220484    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:02.021247 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:02.021215    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00f02541-8772-4fa8-962c-85f970465222" path="/var/lib/kubelet/pods/00f02541-8772-4fa8-962c-85f970465222/volumes"
Jul 28 21:19:02.900545 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:02.900505    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c69f006-5579cbc94d-p566r" podUID="00f02541-8772-4fa8-962c-85f970465222" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.60:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 21:19:06.131266 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:06.131227    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:06.232399 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:06.232352    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:06.388592 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:06.388511    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"]
Jul 28 21:19:06.388758 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:06.388730    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg" podUID="d5fcbcf8-c27c-41b8-9218-73d075d19b2c" containerName="istio-proxy" containerID="cri-o://68d90209495970d00e6938ba2955f76fca56975f9614a1ba7b4710d699681f0f" gracePeriod=30
Jul 28 21:19:06.410485 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:06.410315    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:06.630969 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:06.630933    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:06.743633 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:06.743550    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:06.791452 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:06.791417    2537 patch_prober.go:28] interesting pod/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Jul 28 21:19:06.791591 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:06.791470    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg" podUID="d5fcbcf8-c27c-41b8-9218-73d075d19b2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 21:19:06.848062 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:06.848036    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:06.958338 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:06.958308    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:11.654323 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.654259    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:19:11.745581 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.745553    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6bpl\" (UniqueName: \"kubernetes.io/projected/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-kube-api-access-x6bpl\") pod \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") "
Jul 28 21:19:11.745581 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.745583    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-envoy\") pod \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") "
Jul 28 21:19:11.745798 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.745606    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istiod-ca-cert\") pod \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") "
Jul 28 21:19:11.745798 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.745623    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-data\") pod \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") "
Jul 28 21:19:11.745798 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.745658    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-workload-certs\") pod \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") "
Jul 28 21:19:11.745798 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.745687    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-credential-socket\") pod \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") "
Jul 28 21:19:11.745798 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.745724    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-workload-socket\") pod \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") "
Jul 28 21:19:11.745798 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.745779    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-token\") pod \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") "
Jul 28 21:19:11.746145 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.745833    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-podinfo\") pod \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\" (UID: \"d5fcbcf8-c27c-41b8-9218-73d075d19b2c\") "
Jul 28 21:19:11.746145 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.746069    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5fcbcf8-c27c-41b8-9218-73d075d19b2c" (UID: "d5fcbcf8-c27c-41b8-9218-73d075d19b2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:19:11.746145 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.746069    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5fcbcf8-c27c-41b8-9218-73d075d19b2c" (UID: "d5fcbcf8-c27c-41b8-9218-73d075d19b2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:19:11.746311 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.746154    2537 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istiod-ca-cert\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:19:11.746311 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.746172    2537 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-workload-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:19:11.746311 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.746215    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5fcbcf8-c27c-41b8-9218-73d075d19b2c" (UID: "d5fcbcf8-c27c-41b8-9218-73d075d19b2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:19:11.746311 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.746284    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5fcbcf8-c27c-41b8-9218-73d075d19b2c" (UID: "d5fcbcf8-c27c-41b8-9218-73d075d19b2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:19:11.746601 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.746576    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5fcbcf8-c27c-41b8-9218-73d075d19b2c" (UID: "d5fcbcf8-c27c-41b8-9218-73d075d19b2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:19:11.748147 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.748115    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5fcbcf8-c27c-41b8-9218-73d075d19b2c" (UID: "d5fcbcf8-c27c-41b8-9218-73d075d19b2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:19:11.748256 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.748178    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5fcbcf8-c27c-41b8-9218-73d075d19b2c" (UID: "d5fcbcf8-c27c-41b8-9218-73d075d19b2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 21:19:11.748256 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.748231    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5fcbcf8-c27c-41b8-9218-73d075d19b2c" (UID: "d5fcbcf8-c27c-41b8-9218-73d075d19b2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:19:11.748350 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.748280    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-kube-api-access-x6bpl" (OuterVolumeSpecName: "kube-api-access-x6bpl") pod "d5fcbcf8-c27c-41b8-9218-73d075d19b2c" (UID: "d5fcbcf8-c27c-41b8-9218-73d075d19b2c"). InnerVolumeSpecName "kube-api-access-x6bpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:19:11.846973 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.846947    2537 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-credential-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:19:11.846973 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.846968    2537 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-workload-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:19:11.847129 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.846980    2537 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-token\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:19:11.847129 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.846990    2537 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-podinfo\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:19:11.847129 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.846998    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6bpl\" (UniqueName: \"kubernetes.io/projected/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-kube-api-access-x6bpl\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:19:11.847129 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.847007    2537 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-envoy\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:19:11.847129 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:11.847015    2537 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5fcbcf8-c27c-41b8-9218-73d075d19b2c-istio-data\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:19:12.174057 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.174023    2537 generic.go:356] "Generic (PLEG): container finished" podID="d5fcbcf8-c27c-41b8-9218-73d075d19b2c" containerID="68d90209495970d00e6938ba2955f76fca56975f9614a1ba7b4710d699681f0f" exitCode=0
Jul 28 21:19:12.174217 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.174068    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"
Jul 28 21:19:12.174217 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.174110    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg" event={"ID":"d5fcbcf8-c27c-41b8-9218-73d075d19b2c","Type":"ContainerDied","Data":"68d90209495970d00e6938ba2955f76fca56975f9614a1ba7b4710d699681f0f"}
Jul 28 21:19:12.174217 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.174168    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg" event={"ID":"d5fcbcf8-c27c-41b8-9218-73d075d19b2c","Type":"ContainerDied","Data":"33b7e7357d8878aa57dbfc7e6b43738c9b0249c5808ab4bb32a6d2f0cdec71bb"}
Jul 28 21:19:12.174217 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.174190    2537 scope.go:117] "RemoveContainer" containerID="68d90209495970d00e6938ba2955f76fca56975f9614a1ba7b4710d699681f0f"
Jul 28 21:19:12.183267 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.183244    2537 scope.go:117] "RemoveContainer" containerID="410b99082fd1d19da65fa57288e4930efcc6c8336091cb1eeefd2de012605b2a"
Jul 28 21:19:12.192195 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.192173    2537 scope.go:117] "RemoveContainer" containerID="68d90209495970d00e6938ba2955f76fca56975f9614a1ba7b4710d699681f0f"
Jul 28 21:19:12.192509 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:19:12.192482    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d90209495970d00e6938ba2955f76fca56975f9614a1ba7b4710d699681f0f\": container with ID starting with 68d90209495970d00e6938ba2955f76fca56975f9614a1ba7b4710d699681f0f not found: ID does not exist" containerID="68d90209495970d00e6938ba2955f76fca56975f9614a1ba7b4710d699681f0f"
Jul 28 21:19:12.192619 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.192519    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d90209495970d00e6938ba2955f76fca56975f9614a1ba7b4710d699681f0f"} err="failed to get container status \"68d90209495970d00e6938ba2955f76fca56975f9614a1ba7b4710d699681f0f\": rpc error: code = NotFound desc = could not find container \"68d90209495970d00e6938ba2955f76fca56975f9614a1ba7b4710d699681f0f\": container with ID starting with 68d90209495970d00e6938ba2955f76fca56975f9614a1ba7b4710d699681f0f not found: ID does not exist"
Jul 28 21:19:12.192619 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.192543    2537 scope.go:117] "RemoveContainer" containerID="410b99082fd1d19da65fa57288e4930efcc6c8336091cb1eeefd2de012605b2a"
Jul 28 21:19:12.192953 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:19:12.192930    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410b99082fd1d19da65fa57288e4930efcc6c8336091cb1eeefd2de012605b2a\": container with ID starting with 410b99082fd1d19da65fa57288e4930efcc6c8336091cb1eeefd2de012605b2a not found: ID does not exist" containerID="410b99082fd1d19da65fa57288e4930efcc6c8336091cb1eeefd2de012605b2a"
Jul 28 21:19:12.193058 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.192961    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410b99082fd1d19da65fa57288e4930efcc6c8336091cb1eeefd2de012605b2a"} err="failed to get container status \"410b99082fd1d19da65fa57288e4930efcc6c8336091cb1eeefd2de012605b2a\": rpc error: code = NotFound desc = could not find container \"410b99082fd1d19da65fa57288e4930efcc6c8336091cb1eeefd2de012605b2a\": container with ID starting with 410b99082fd1d19da65fa57288e4930efcc6c8336091cb1eeefd2de012605b2a not found: ID does not exist"
Jul 28 21:19:12.195149 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.195129    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"]
Jul 28 21:19:12.198679 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.198661    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7c69f006-openshift-default-6b5c8cb756-khwkg"]
Jul 28 21:19:12.755325 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.755291    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:12.810561 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.810534    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:12.881664 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.881636    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:12.980174 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:12.980140    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:13.084050 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:13.084009    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:14.021546 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:14.021514    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5fcbcf8-c27c-41b8-9218-73d075d19b2c" path="/var/lib/kubelet/pods/d5fcbcf8-c27c-41b8-9218-73d075d19b2c/volumes"
Jul 28 21:19:19.039018 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:19.038981    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:19.139788 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:19.139752    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:25.888775 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:25.888739    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:26.171089 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.171017    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"]
Jul 28 21:19:26.171414 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.171402    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f02541-8772-4fa8-962c-85f970465222" containerName="maas-api"
Jul 28 21:19:26.171462 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.171415    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f02541-8772-4fa8-962c-85f970465222" containerName="maas-api"
Jul 28 21:19:26.171462 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.171426    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5fcbcf8-c27c-41b8-9218-73d075d19b2c" containerName="istio-proxy"
Jul 28 21:19:26.171462 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.171432    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5fcbcf8-c27c-41b8-9218-73d075d19b2c" containerName="istio-proxy"
Jul 28 21:19:26.171462 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.171438    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5fcbcf8-c27c-41b8-9218-73d075d19b2c" containerName="istio-proxy"
Jul 28 21:19:26.171462 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.171443    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5fcbcf8-c27c-41b8-9218-73d075d19b2c" containerName="istio-proxy"
Jul 28 21:19:26.171616 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.171496    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5fcbcf8-c27c-41b8-9218-73d075d19b2c" containerName="istio-proxy"
Jul 28 21:19:26.171616 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.171504    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="00f02541-8772-4fa8-962c-85f970465222" containerName="maas-api"
Jul 28 21:19:26.174528 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.174512    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.177203 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.177185    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6b0b6c51-openshift-default-dockercfg-dn4t9\""
Jul 28 21:19:26.185086 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.185060    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"]
Jul 28 21:19:26.197266 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.197240    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:26.268335 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.268305    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-envoy\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.268518 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.268353    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-credential-socket\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.268518 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.268375    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istiod-ca-cert\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.268518 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.268501    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-podinfo\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.268706 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.268538    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-674hw\" (UniqueName: \"kubernetes.io/projected/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-kube-api-access-674hw\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.268706 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.268675    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-token\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.268810 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.268753    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-workload-socket\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.268810 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.268790    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-data\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.268949 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.268823    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-workload-certs\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.369952 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.369919    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-workload-certs\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.369952 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.369960    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-envoy\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.370146 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.370009    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-credential-socket\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.370146 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.370056    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istiod-ca-cert\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.370146 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.370127    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-podinfo\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.370284 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.370153    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-674hw\" (UniqueName: \"kubernetes.io/projected/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-kube-api-access-674hw\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.370284 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.370199    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-token\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.370284 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.370235    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-workload-socket\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.370440 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.370362    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-workload-certs\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.370440 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.370396    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-credential-socket\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.370547 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.370469    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-data\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.370547 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.370501    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-workload-socket\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.370790 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.370771    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-data\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.370997 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.370977    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istiod-ca-cert\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.372327 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.372305    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-envoy\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.372580 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.372562    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-podinfo\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.378344 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.378319    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-674hw\" (UniqueName: \"kubernetes.io/projected/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-kube-api-access-674hw\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.378903 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.378838    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-token\") pod \"e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") " pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.485684 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.485617    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:26.620315 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.620290    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"]
Jul 28 21:19:26.622500 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:19:26.622475    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0724c864_8d30_4430_b24d_9b9f4b6ea6cc.slice/crio-508223b504ce495c5c2f7b8bc58f159051ffcfcb892db159f44be6a3098ba912 WatchSource:0}: Error finding container 508223b504ce495c5c2f7b8bc58f159051ffcfcb892db159f44be6a3098ba912: Status 404 returned error can't find the container with id 508223b504ce495c5c2f7b8bc58f159051ffcfcb892db159f44be6a3098ba912
Jul 28 21:19:26.624772 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.624736    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:19:26.624910 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.624805    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:19:26.624910 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.624846    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:19:26.767974 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:26.767884    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:27.257107 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:27.257063    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq" event={"ID":"0724c864-8d30-4430-b24d-9b9f4b6ea6cc","Type":"ContainerStarted","Data":"bacaaff0edcdc5ce425bbfbe3945154cde4457ef61fc7536fe01a1a13ce6a2dd"}
Jul 28 21:19:27.257474 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:27.257160    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq" event={"ID":"0724c864-8d30-4430-b24d-9b9f4b6ea6cc","Type":"ContainerStarted","Data":"508223b504ce495c5c2f7b8bc58f159051ffcfcb892db159f44be6a3098ba912"}
Jul 28 21:19:27.280067 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:27.280015    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq" podStartSLOduration=1.280001223 podStartE2EDuration="1.280001223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:19:27.277764977 +0000 UTC m=+1927.895346879" watchObservedRunningTime="2026-07-28 21:19:27.280001223 +0000 UTC m=+1927.897583125"
Jul 28 21:19:27.486212 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:27.486182    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:27.490803 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:27.490781    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:28.261406 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:28.261371    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:28.262342 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:28.262321    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:29.135730 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.135696    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:29.213737 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.213700    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:29.215050 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.215028    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"]
Jul 28 21:19:29.215512 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.215498    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5fcbcf8-c27c-41b8-9218-73d075d19b2c" containerName="istio-proxy"
Jul 28 21:19:29.218789 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.218769    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:19:29.221324 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.221302    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6b0b6c51-serving-cert\""
Jul 28 21:19:29.229184 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.229158    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"]
Jul 28 21:19:29.274184 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.274139    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:29.294660 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.294633    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f33b2b1-1345-4691-a967-13b4c6d1964b-openshift-service-ca\") pod \"maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x\" (UID: \"6f33b2b1-1345-4691-a967-13b4c6d1964b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:19:29.294776 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.294732    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f33b2b1-1345-4691-a967-13b4c6d1964b-maas-api-tls\") pod \"maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x\" (UID: \"6f33b2b1-1345-4691-a967-13b4c6d1964b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:19:29.294776 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.294756    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtz97\" (UniqueName: \"kubernetes.io/projected/6f33b2b1-1345-4691-a967-13b4c6d1964b-kube-api-access-gtz97\") pod \"maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x\" (UID: \"6f33b2b1-1345-4691-a967-13b4c6d1964b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:19:29.395419 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.395336    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f33b2b1-1345-4691-a967-13b4c6d1964b-openshift-service-ca\") pod \"maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x\" (UID: \"6f33b2b1-1345-4691-a967-13b4c6d1964b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:19:29.395586 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.395476    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f33b2b1-1345-4691-a967-13b4c6d1964b-maas-api-tls\") pod \"maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x\" (UID: \"6f33b2b1-1345-4691-a967-13b4c6d1964b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:19:29.395586 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.395509    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtz97\" (UniqueName: \"kubernetes.io/projected/6f33b2b1-1345-4691-a967-13b4c6d1964b-kube-api-access-gtz97\") pod \"maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x\" (UID: \"6f33b2b1-1345-4691-a967-13b4c6d1964b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:19:29.395718 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:19:29.395638    2537 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-serving-cert: secret "maas-api-e2e-mt-6b0b6c51-serving-cert" not found
Jul 28 21:19:29.395773 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:19:29.395721    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f33b2b1-1345-4691-a967-13b4c6d1964b-maas-api-tls podName:6f33b2b1-1345-4691-a967-13b4c6d1964b nodeName:}" failed. No retries permitted until 2026-07-28 21:19:29.895698285 +0000 UTC m=+1930.513280168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f33b2b1-1345-4691-a967-13b4c6d1964b-maas-api-tls") pod "maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x" (UID: "6f33b2b1-1345-4691-a967-13b4c6d1964b") : secret "maas-api-e2e-mt-6b0b6c51-serving-cert" not found
Jul 28 21:19:29.396128 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.396109    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f33b2b1-1345-4691-a967-13b4c6d1964b-openshift-service-ca\") pod \"maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x\" (UID: \"6f33b2b1-1345-4691-a967-13b4c6d1964b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:19:29.404649 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.404624    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtz97\" (UniqueName: \"kubernetes.io/projected/6f33b2b1-1345-4691-a967-13b4c6d1964b-kube-api-access-gtz97\") pod \"maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x\" (UID: \"6f33b2b1-1345-4691-a967-13b4c6d1964b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:19:29.517063 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.517017    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:29.710989 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.710895    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:29.815055 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.815018    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:19:29.899494 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.899462    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f33b2b1-1345-4691-a967-13b4c6d1964b-maas-api-tls\") pod \"maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x\" (UID: \"6f33b2b1-1345-4691-a967-13b4c6d1964b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:19:29.901883 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:29.901850    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f33b2b1-1345-4691-a967-13b4c6d1964b-maas-api-tls\") pod \"maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x\" (UID: \"6f33b2b1-1345-4691-a967-13b4c6d1964b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:19:30.131598 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:30.131565    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:19:30.266800 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:30.266774    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"]
Jul 28 21:19:30.268442 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:19:30.268415    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f33b2b1_1345_4691_a967_13b4c6d1964b.slice/crio-d608b3855223a9abe3aa1eca083f64deee3b6d0ba9d59db82d4bd7a1ad724604 WatchSource:0}: Error finding container d608b3855223a9abe3aa1eca083f64deee3b6d0ba9d59db82d4bd7a1ad724604: Status 404 returned error can't find the container with id d608b3855223a9abe3aa1eca083f64deee3b6d0ba9d59db82d4bd7a1ad724604
Jul 28 21:19:30.270097 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:30.270079    2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 21:19:31.279165 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:31.279128    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x" event={"ID":"6f33b2b1-1345-4691-a967-13b4c6d1964b","Type":"ContainerStarted","Data":"7b45b8175cdc9dff250bf760a51008859efc9720a728d1ed13d73707513063af"}
Jul 28 21:19:31.279589 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:31.279196    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:19:31.279589 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:31.279220    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x" event={"ID":"6f33b2b1-1345-4691-a967-13b4c6d1964b","Type":"ContainerStarted","Data":"d608b3855223a9abe3aa1eca083f64deee3b6d0ba9d59db82d4bd7a1ad724604"}
Jul 28 21:19:31.298716 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:31.298661    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x" podStartSLOduration=1.850449601 podStartE2EDuration="2.298646479s" totalImagesPullingTime="448.196878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:19:31.297259598 +0000 UTC m=+1931.914841503" watchObservedRunningTime="2026-07-28 21:19:31.298646479 +0000 UTC m=+1931.916228381"
Jul 28 21:19:37.289597 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:37.289563    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:19:42.338773 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:42.338739    2537 generic.go:356] "Generic (PLEG): container finished" podID="0724c864-8d30-4430-b24d-9b9f4b6ea6cc" containerID="bacaaff0edcdc5ce425bbfbe3945154cde4457ef61fc7536fe01a1a13ce6a2dd" exitCode=0
Jul 28 21:19:42.339189 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:42.338814    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq" event={"ID":"0724c864-8d30-4430-b24d-9b9f4b6ea6cc","Type":"ContainerDied","Data":"bacaaff0edcdc5ce425bbfbe3945154cde4457ef61fc7536fe01a1a13ce6a2dd"}
Jul 28 21:19:42.339236 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:42.339218    2537 scope.go:117] "RemoveContainer" containerID="bacaaff0edcdc5ce425bbfbe3945154cde4457ef61fc7536fe01a1a13ce6a2dd"
Jul 28 21:19:42.340305 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:42.340274    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:19:42.340377 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:42.340360    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:19:42.340425 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:42.340408    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:19:43.262231 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:43.262193    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:43.348716 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:43.348682    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq" event={"ID":"0724c864-8d30-4430-b24d-9b9f4b6ea6cc","Type":"ContainerStarted","Data":"eac70c7e714d12afa2dd18f1292d2ee76a770efc871a7e87116f96a1a9ef7daa"}
Jul 28 21:19:43.486587 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:43.486548    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:43.487814 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:43.487789    2537 patch_prober.go:28] interesting pod/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 28 21:19:43.487943 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:43.487835    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq" podUID="0724c864-8d30-4430-b24d-9b9f4b6ea6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 28 21:19:44.486199 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:44.486158    2537 patch_prober.go:28] interesting pod/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 28 21:19:44.486602 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:44.486216    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq" podUID="0724c864-8d30-4430-b24d-9b9f4b6ea6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 28 21:19:45.485997 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:45.485958    2537 patch_prober.go:28] interesting pod/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 28 21:19:45.486170 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:45.486029    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq" podUID="0724c864-8d30-4430-b24d-9b9f4b6ea6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 28 21:19:46.486702 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:46.486669    2537 patch_prober.go:28] interesting pod/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 28 21:19:46.487130 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:46.486719    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq" podUID="0724c864-8d30-4430-b24d-9b9f4b6ea6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 28 21:19:47.489975 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:47.489941    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:47.490413 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:47.490215    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:47.490949 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:47.490931    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:19:54.091967 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:19:54.091929    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:20:11.459117 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:11.459080    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:20:18.778552 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:18.778467    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"]
Jul 28 21:20:18.782460 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:18.782442    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:18.785351 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:18.785327    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6b0b6c51\"/\"openshift-service-ca.crt\""
Jul 28 21:20:18.786794 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:18.786775    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6b0b6c51\"/\"e2e-lifecycle-model-6b0b6c51-kserve-self-signed-certs\""
Jul 28 21:20:18.786926 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:18.786790    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6b0b6c51\"/\"default-dockercfg-6d5wl\""
Jul 28 21:20:18.786926 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:18.786811    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6b0b6c51\"/\"kube-root-ca.crt\""
Jul 28 21:20:18.790387 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:18.790364    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"]
Jul 28 21:20:18.804785 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:18.804754    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:20:18.933531 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:18.933499    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-model-cache\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:18.933531 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:18.933534    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx4g2\" (UniqueName: \"kubernetes.io/projected/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-kube-api-access-lx4g2\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:18.933775 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:18.933624    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-tmp-dir\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:18.933775 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:18.933654    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-dshm\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:18.933775 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:18.933710    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-home\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:18.933933 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:18.933781    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-tls-certs\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:19.035259 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.035177    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-model-cache\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:19.035259 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.035216    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx4g2\" (UniqueName: \"kubernetes.io/projected/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-kube-api-access-lx4g2\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:19.035485 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.035283    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-tmp-dir\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:19.035485 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.035327    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-dshm\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:19.035485 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.035364    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-home\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:19.035485 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.035394    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-tls-certs\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:19.035685 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.035629    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-model-cache\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:19.035685 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.035644    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-tmp-dir\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:19.035797 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.035710    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-home\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:19.037512 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.037479    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-dshm\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:19.037908 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.037889    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-tls-certs\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:19.047122 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.047098    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx4g2\" (UniqueName: \"kubernetes.io/projected/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-kube-api-access-lx4g2\") pod \"e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") " pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:19.093226 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.093204    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:19.233461 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.233436    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"]
Jul 28 21:20:19.235877 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:20:19.235830    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd657d9e_2c36_4ef8_8fba_7c4a506e3e42.slice/crio-024ffdaa8bff0108a2774dc6cf17463ae985c28c27c5d4a05ec2b065d76ee7f9 WatchSource:0}: Error finding container 024ffdaa8bff0108a2774dc6cf17463ae985c28c27c5d4a05ec2b065d76ee7f9: Status 404 returned error can't find the container with id 024ffdaa8bff0108a2774dc6cf17463ae985c28c27c5d4a05ec2b065d76ee7f9
Jul 28 21:20:19.522511 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.522474    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l" event={"ID":"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42","Type":"ContainerStarted","Data":"bb9f614bb80db7160ab5f07ca21ecd8ba254ce5d68bb26b2d8433bca8f0aac04"}
Jul 28 21:20:19.522670 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.522522    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l" event={"ID":"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42","Type":"ContainerStarted","Data":"024ffdaa8bff0108a2774dc6cf17463ae985c28c27c5d4a05ec2b065d76ee7f9"}
Jul 28 21:20:19.543811 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:19.543727    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l" podStartSLOduration=1.543712503 podStartE2EDuration="1.543712503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:20:19.543019372 +0000 UTC m=+1980.160601276" watchObservedRunningTime="2026-07-28 21:20:19.543712503 +0000 UTC m=+1980.161294406"
Jul 28 21:20:29.093678 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:29.093642    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:29.094179 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:29.093701    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:29.105960 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:29.105934    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:29.589560 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:29.589530    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:20:54.070178 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.070137    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"]
Jul 28 21:20:54.070701 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.070397    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x" podUID="6f33b2b1-1345-4691-a967-13b4c6d1964b" containerName="maas-api" containerID="cri-o://7b45b8175cdc9dff250bf760a51008859efc9720a728d1ed13d73707513063af" gracePeriod=30
Jul 28 21:20:54.094636 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.094608    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:20:54.234194 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.234147    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:20:54.310907 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.310876    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:20:54.321556 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.321500    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:20:54.382073 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.382042    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:20:54.437888 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.437825    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f33b2b1-1345-4691-a967-13b4c6d1964b-maas-api-tls\") pod \"6f33b2b1-1345-4691-a967-13b4c6d1964b\" (UID: \"6f33b2b1-1345-4691-a967-13b4c6d1964b\") "
Jul 28 21:20:54.438077 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.437948    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f33b2b1-1345-4691-a967-13b4c6d1964b-openshift-service-ca\") pod \"6f33b2b1-1345-4691-a967-13b4c6d1964b\" (UID: \"6f33b2b1-1345-4691-a967-13b4c6d1964b\") "
Jul 28 21:20:54.438077 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.438003    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtz97\" (UniqueName: \"kubernetes.io/projected/6f33b2b1-1345-4691-a967-13b4c6d1964b-kube-api-access-gtz97\") pod \"6f33b2b1-1345-4691-a967-13b4c6d1964b\" (UID: \"6f33b2b1-1345-4691-a967-13b4c6d1964b\") "
Jul 28 21:20:54.438299 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.438275    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f33b2b1-1345-4691-a967-13b4c6d1964b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f33b2b1-1345-4691-a967-13b4c6d1964b" (UID: "6f33b2b1-1345-4691-a967-13b4c6d1964b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:20:54.439926 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.439905    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f33b2b1-1345-4691-a967-13b4c6d1964b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f33b2b1-1345-4691-a967-13b4c6d1964b" (UID: "6f33b2b1-1345-4691-a967-13b4c6d1964b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:20:54.440034 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.440011    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f33b2b1-1345-4691-a967-13b4c6d1964b-kube-api-access-gtz97" (OuterVolumeSpecName: "kube-api-access-gtz97") pod "6f33b2b1-1345-4691-a967-13b4c6d1964b" (UID: "6f33b2b1-1345-4691-a967-13b4c6d1964b"). InnerVolumeSpecName "kube-api-access-gtz97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:20:54.539260 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.539229    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f33b2b1-1345-4691-a967-13b4c6d1964b-openshift-service-ca\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:20:54.539260 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.539254    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtz97\" (UniqueName: \"kubernetes.io/projected/6f33b2b1-1345-4691-a967-13b4c6d1964b-kube-api-access-gtz97\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:20:54.539260 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.539263    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f33b2b1-1345-4691-a967-13b4c6d1964b-maas-api-tls\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:20:54.713262 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.713175    2537 generic.go:356] "Generic (PLEG): container finished" podID="6f33b2b1-1345-4691-a967-13b4c6d1964b" containerID="7b45b8175cdc9dff250bf760a51008859efc9720a728d1ed13d73707513063af" exitCode=0
Jul 28 21:20:54.713262 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.713225    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"
Jul 28 21:20:54.713432 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.713264    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x" event={"ID":"6f33b2b1-1345-4691-a967-13b4c6d1964b","Type":"ContainerDied","Data":"7b45b8175cdc9dff250bf760a51008859efc9720a728d1ed13d73707513063af"}
Jul 28 21:20:54.713432 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.713322    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x" event={"ID":"6f33b2b1-1345-4691-a967-13b4c6d1964b","Type":"ContainerDied","Data":"d608b3855223a9abe3aa1eca083f64deee3b6d0ba9d59db82d4bd7a1ad724604"}
Jul 28 21:20:54.713432 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.713351    2537 scope.go:117] "RemoveContainer" containerID="7b45b8175cdc9dff250bf760a51008859efc9720a728d1ed13d73707513063af"
Jul 28 21:20:54.722521 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.722505    2537 scope.go:117] "RemoveContainer" containerID="7b45b8175cdc9dff250bf760a51008859efc9720a728d1ed13d73707513063af"
Jul 28 21:20:54.722775 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:20:54.722757    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b45b8175cdc9dff250bf760a51008859efc9720a728d1ed13d73707513063af\": container with ID starting with 7b45b8175cdc9dff250bf760a51008859efc9720a728d1ed13d73707513063af not found: ID does not exist" containerID="7b45b8175cdc9dff250bf760a51008859efc9720a728d1ed13d73707513063af"
Jul 28 21:20:54.722847 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.722784    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b45b8175cdc9dff250bf760a51008859efc9720a728d1ed13d73707513063af"} err="failed to get container status \"7b45b8175cdc9dff250bf760a51008859efc9720a728d1ed13d73707513063af\": rpc error: code = NotFound desc = could not find container \"7b45b8175cdc9dff250bf760a51008859efc9720a728d1ed13d73707513063af\": container with ID starting with 7b45b8175cdc9dff250bf760a51008859efc9720a728d1ed13d73707513063af not found: ID does not exist"
Jul 28 21:20:54.753479 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.753455    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"]
Jul 28 21:20:54.763298 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:54.763275    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b0b6c51-5cc7664748-sqv2x"]
Jul 28 21:20:56.021089 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:56.021043    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f33b2b1-1345-4691-a967-13b4c6d1964b" path="/var/lib/kubelet/pods/6f33b2b1-1345-4691-a967-13b4c6d1964b/volumes"
Jul 28 21:20:57.453056 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:20:57.453014    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:21:02.435445 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:02.435408    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"]
Jul 28 21:21:02.529521 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:02.529485    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:21:17.454697 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.454656    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l" podUID="dd657d9e-2c36-4ef8-8fba-7c4a506e3e42" containerName="main" containerID="cri-o://bb9f614bb80db7160ab5f07ca21ecd8ba254ce5d68bb26b2d8433bca8f0aac04" gracePeriod=45
Jul 28 21:21:17.708349 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.708298    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:21:17.823985 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.823946    2537 generic.go:356] "Generic (PLEG): container finished" podID="dd657d9e-2c36-4ef8-8fba-7c4a506e3e42" containerID="bb9f614bb80db7160ab5f07ca21ecd8ba254ce5d68bb26b2d8433bca8f0aac04" exitCode=0
Jul 28 21:21:17.824161 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.824001    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"
Jul 28 21:21:17.824161 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.824040    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l" event={"ID":"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42","Type":"ContainerDied","Data":"bb9f614bb80db7160ab5f07ca21ecd8ba254ce5d68bb26b2d8433bca8f0aac04"}
Jul 28 21:21:17.824161 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.824084    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l" event={"ID":"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42","Type":"ContainerDied","Data":"024ffdaa8bff0108a2774dc6cf17463ae985c28c27c5d4a05ec2b065d76ee7f9"}
Jul 28 21:21:17.824161 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.824106    2537 scope.go:117] "RemoveContainer" containerID="bb9f614bb80db7160ab5f07ca21ecd8ba254ce5d68bb26b2d8433bca8f0aac04"
Jul 28 21:21:17.832749 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.832727    2537 scope.go:117] "RemoveContainer" containerID="bb9f614bb80db7160ab5f07ca21ecd8ba254ce5d68bb26b2d8433bca8f0aac04"
Jul 28 21:21:17.833087 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:21:17.833053    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9f614bb80db7160ab5f07ca21ecd8ba254ce5d68bb26b2d8433bca8f0aac04\": container with ID starting with bb9f614bb80db7160ab5f07ca21ecd8ba254ce5d68bb26b2d8433bca8f0aac04 not found: ID does not exist" containerID="bb9f614bb80db7160ab5f07ca21ecd8ba254ce5d68bb26b2d8433bca8f0aac04"
Jul 28 21:21:17.833198 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.833094    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9f614bb80db7160ab5f07ca21ecd8ba254ce5d68bb26b2d8433bca8f0aac04"} err="failed to get container status \"bb9f614bb80db7160ab5f07ca21ecd8ba254ce5d68bb26b2d8433bca8f0aac04\": rpc error: code = NotFound desc = could not find container \"bb9f614bb80db7160ab5f07ca21ecd8ba254ce5d68bb26b2d8433bca8f0aac04\": container with ID starting with bb9f614bb80db7160ab5f07ca21ecd8ba254ce5d68bb26b2d8433bca8f0aac04 not found: ID does not exist"
Jul 28 21:21:17.842372 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.842354    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-dshm\") pod \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") "
Jul 28 21:21:17.842469 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.842408    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx4g2\" (UniqueName: \"kubernetes.io/projected/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-kube-api-access-lx4g2\") pod \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") "
Jul 28 21:21:17.842469 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.842426    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-tls-certs\") pod \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") "
Jul 28 21:21:17.842469 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.842456    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-model-cache\") pod \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") "
Jul 28 21:21:17.842647 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.842479    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-tmp-dir\") pod \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") "
Jul 28 21:21:17.842647 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.842500    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-home\") pod \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\" (UID: \"dd657d9e-2c36-4ef8-8fba-7c4a506e3e42\") "
Jul 28 21:21:17.842777 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.842745    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd657d9e-2c36-4ef8-8fba-7c4a506e3e42" (UID: "dd657d9e-2c36-4ef8-8fba-7c4a506e3e42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:21:17.842833 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.842761    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd657d9e-2c36-4ef8-8fba-7c4a506e3e42" (UID: "dd657d9e-2c36-4ef8-8fba-7c4a506e3e42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:21:17.842893 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.842839    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-home" (OuterVolumeSpecName: "home") pod "dd657d9e-2c36-4ef8-8fba-7c4a506e3e42" (UID: "dd657d9e-2c36-4ef8-8fba-7c4a506e3e42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:21:17.844608 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.844590    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-dshm" (OuterVolumeSpecName: "dshm") pod "dd657d9e-2c36-4ef8-8fba-7c4a506e3e42" (UID: "dd657d9e-2c36-4ef8-8fba-7c4a506e3e42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:21:17.844678 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.844604    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd657d9e-2c36-4ef8-8fba-7c4a506e3e42" (UID: "dd657d9e-2c36-4ef8-8fba-7c4a506e3e42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:21:17.844678 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.844627    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-kube-api-access-lx4g2" (OuterVolumeSpecName: "kube-api-access-lx4g2") pod "dd657d9e-2c36-4ef8-8fba-7c4a506e3e42" (UID: "dd657d9e-2c36-4ef8-8fba-7c4a506e3e42"). InnerVolumeSpecName "kube-api-access-lx4g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:21:17.943519 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.943492    2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-tmp-dir\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:21:17.943519 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.943518    2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-home\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:21:17.943519 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.943526    2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-dshm\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:21:17.943751 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.943535    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx4g2\" (UniqueName: \"kubernetes.io/projected/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-kube-api-access-lx4g2\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:21:17.943751 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.943546    2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-tls-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:21:17.943751 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:17.943554    2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42-model-cache\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:21:18.140160 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:18.140128    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"]
Jul 28 21:21:18.144121 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:18.144096    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6b0b6c51/e2e-lifecycle-model-6b0b6c51-kserve-5789bcf84c-d6l7l"]
Jul 28 21:21:20.022197 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:20.022159    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd657d9e-2c36-4ef8-8fba-7c4a506e3e42" path="/var/lib/kubelet/pods/dd657d9e-2c36-4ef8-8fba-7c4a506e3e42/volumes"
Jul 28 21:21:39.604344 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:39.604305    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:21:39.775298 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:39.775270    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:21:45.497746 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:21:45.497666    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:00.116460 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:00.116421    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:00.216793 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:00.216755    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:00.360324 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:00.360292    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"]
Jul 28 21:22:00.360572 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:00.360547    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq" podUID="0724c864-8d30-4430-b24d-9b9f4b6ea6cc" containerName="istio-proxy" containerID="cri-o://eac70c7e714d12afa2dd18f1292d2ee76a770efc871a7e87116f96a1a9ef7daa" gracePeriod=30
Jul 28 21:22:00.383475 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:00.383404    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:00.605995 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:00.605961    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:01.657361 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:01.657323    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:01.756117 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:01.756081    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:01.852848 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:01.852818    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:02.492384 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:02.492335    2537 patch_prober.go:28] interesting pod/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 21:22:02.492581 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:02.492403    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq" podUID="0724c864-8d30-4430-b24d-9b9f4b6ea6cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 21:22:02.525515 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:02.525485    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:02.624846 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:02.624806    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:02.720972 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:02.720933    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:04.044051 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:04.044019    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:05.621877 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.621772    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:22:05.751521 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.751487    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-podinfo\") pod \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") "
Jul 28 21:22:05.751521 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.751529    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-workload-socket\") pod \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") "
Jul 28 21:22:05.751783 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.751564    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-674hw\" (UniqueName: \"kubernetes.io/projected/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-kube-api-access-674hw\") pod \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") "
Jul 28 21:22:05.751783 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.751676    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-envoy\") pod \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") "
Jul 28 21:22:05.751783 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.751706    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-data\") pod \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") "
Jul 28 21:22:05.751783 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.751746    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-credential-socket\") pod \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") "
Jul 28 21:22:05.752045 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.751797    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-token\") pod \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") "
Jul 28 21:22:05.752045 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.751817    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0724c864-8d30-4430-b24d-9b9f4b6ea6cc" (UID: "0724c864-8d30-4430-b24d-9b9f4b6ea6cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:22:05.752045 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.751830    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-workload-certs\") pod \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") "
Jul 28 21:22:05.752045 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.751940    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istiod-ca-cert\") pod \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\" (UID: \"0724c864-8d30-4430-b24d-9b9f4b6ea6cc\") "
Jul 28 21:22:05.762324 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.752707    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-data" (OuterVolumeSpecName: "istio-data") pod "0724c864-8d30-4430-b24d-9b9f4b6ea6cc" (UID: "0724c864-8d30-4430-b24d-9b9f4b6ea6cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:22:05.762324 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.752960    2537 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-workload-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:22:05.762324 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.752981    2537 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-data\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:22:05.762324 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.753900    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0724c864-8d30-4430-b24d-9b9f4b6ea6cc" (UID: "0724c864-8d30-4430-b24d-9b9f4b6ea6cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:22:05.762324 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.754082    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0724c864-8d30-4430-b24d-9b9f4b6ea6cc" (UID: "0724c864-8d30-4430-b24d-9b9f4b6ea6cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:22:05.762324 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.754120    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0724c864-8d30-4430-b24d-9b9f4b6ea6cc" (UID: "0724c864-8d30-4430-b24d-9b9f4b6ea6cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:22:05.762324 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.754628    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-kube-api-access-674hw" (OuterVolumeSpecName: "kube-api-access-674hw") pod "0724c864-8d30-4430-b24d-9b9f4b6ea6cc" (UID: "0724c864-8d30-4430-b24d-9b9f4b6ea6cc"). InnerVolumeSpecName "kube-api-access-674hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:22:05.763019 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.762811    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0724c864-8d30-4430-b24d-9b9f4b6ea6cc" (UID: "0724c864-8d30-4430-b24d-9b9f4b6ea6cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 21:22:05.763225 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.763073    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0724c864-8d30-4430-b24d-9b9f4b6ea6cc" (UID: "0724c864-8d30-4430-b24d-9b9f4b6ea6cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:22:05.763225 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.763084    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-token" (OuterVolumeSpecName: "istio-token") pod "0724c864-8d30-4430-b24d-9b9f4b6ea6cc" (UID: "0724c864-8d30-4430-b24d-9b9f4b6ea6cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:22:05.853710 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.853667    2537 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-workload-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:22:05.853710 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.853704    2537 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istiod-ca-cert\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:22:05.853710 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.853713    2537 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-podinfo\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:22:05.853979 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.853722    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-674hw\" (UniqueName: \"kubernetes.io/projected/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-kube-api-access-674hw\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:22:05.853979 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.853735    2537 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-envoy\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:22:05.853979 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.853743    2537 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-credential-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:22:05.853979 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:05.853751    2537 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0724c864-8d30-4430-b24d-9b9f4b6ea6cc-istio-token\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:22:06.058820 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:06.058788    2537 generic.go:356] "Generic (PLEG): container finished" podID="0724c864-8d30-4430-b24d-9b9f4b6ea6cc" containerID="eac70c7e714d12afa2dd18f1292d2ee76a770efc871a7e87116f96a1a9ef7daa" exitCode=0
Jul 28 21:22:06.058992 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:06.058883    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"
Jul 28 21:22:06.058992 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:06.058882    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq" event={"ID":"0724c864-8d30-4430-b24d-9b9f4b6ea6cc","Type":"ContainerDied","Data":"eac70c7e714d12afa2dd18f1292d2ee76a770efc871a7e87116f96a1a9ef7daa"}
Jul 28 21:22:06.059075 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:06.059011    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq" event={"ID":"0724c864-8d30-4430-b24d-9b9f4b6ea6cc","Type":"ContainerDied","Data":"508223b504ce495c5c2f7b8bc58f159051ffcfcb892db159f44be6a3098ba912"}
Jul 28 21:22:06.059075 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:06.059042    2537 scope.go:117] "RemoveContainer" containerID="eac70c7e714d12afa2dd18f1292d2ee76a770efc871a7e87116f96a1a9ef7daa"
Jul 28 21:22:06.068509 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:06.068491    2537 scope.go:117] "RemoveContainer" containerID="bacaaff0edcdc5ce425bbfbe3945154cde4457ef61fc7536fe01a1a13ce6a2dd"
Jul 28 21:22:06.076771 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:06.076733    2537 scope.go:117] "RemoveContainer" containerID="eac70c7e714d12afa2dd18f1292d2ee76a770efc871a7e87116f96a1a9ef7daa"
Jul 28 21:22:06.077092 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:22:06.077059    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eac70c7e714d12afa2dd18f1292d2ee76a770efc871a7e87116f96a1a9ef7daa\": container with ID starting with eac70c7e714d12afa2dd18f1292d2ee76a770efc871a7e87116f96a1a9ef7daa not found: ID does not exist" containerID="eac70c7e714d12afa2dd18f1292d2ee76a770efc871a7e87116f96a1a9ef7daa"
Jul 28 21:22:06.077179 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:06.077103    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eac70c7e714d12afa2dd18f1292d2ee76a770efc871a7e87116f96a1a9ef7daa"} err="failed to get container status \"eac70c7e714d12afa2dd18f1292d2ee76a770efc871a7e87116f96a1a9ef7daa\": rpc error: code = NotFound desc = could not find container \"eac70c7e714d12afa2dd18f1292d2ee76a770efc871a7e87116f96a1a9ef7daa\": container with ID starting with eac70c7e714d12afa2dd18f1292d2ee76a770efc871a7e87116f96a1a9ef7daa not found: ID does not exist"
Jul 28 21:22:06.077179 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:06.077128    2537 scope.go:117] "RemoveContainer" containerID="bacaaff0edcdc5ce425bbfbe3945154cde4457ef61fc7536fe01a1a13ce6a2dd"
Jul 28 21:22:06.077409 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:22:06.077391    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bacaaff0edcdc5ce425bbfbe3945154cde4457ef61fc7536fe01a1a13ce6a2dd\": container with ID starting with bacaaff0edcdc5ce425bbfbe3945154cde4457ef61fc7536fe01a1a13ce6a2dd not found: ID does not exist" containerID="bacaaff0edcdc5ce425bbfbe3945154cde4457ef61fc7536fe01a1a13ce6a2dd"
Jul 28 21:22:06.077459 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:06.077415    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bacaaff0edcdc5ce425bbfbe3945154cde4457ef61fc7536fe01a1a13ce6a2dd"} err="failed to get container status \"bacaaff0edcdc5ce425bbfbe3945154cde4457ef61fc7536fe01a1a13ce6a2dd\": rpc error: code = NotFound desc = could not find container \"bacaaff0edcdc5ce425bbfbe3945154cde4457ef61fc7536fe01a1a13ce6a2dd\": container with ID starting with bacaaff0edcdc5ce425bbfbe3945154cde4457ef61fc7536fe01a1a13ce6a2dd not found: ID does not exist"
Jul 28 21:22:06.099149 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:06.099122    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"]
Jul 28 21:22:06.109493 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:06.109470    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b0b6c51-openshift-default-5c654f77b-2xqpq"]
Jul 28 21:22:08.021545 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:08.021513    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0724c864-8d30-4430-b24d-9b9f4b6ea6cc" path="/var/lib/kubelet/pods/0724c864-8d30-4430-b24d-9b9f4b6ea6cc/volumes"
Jul 28 21:22:10.175727 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:10.175694    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:10.276006 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:10.275972    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:10.829545 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:10.829507    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:16.797657 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:16.797623    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:16.906780 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:16.906739    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:17.003781 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:17.003739    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:20.005418 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:20.005386    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:22:20.009528 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:20.009505    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:22:27.536718 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:27.536679    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:27.878431 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:27.878384    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:48.797749 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:48.797708    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:55.094344 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.094307    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:55.195245 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.195209    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:55.428330 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.428221    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:55.700720 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.700648    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"]
Jul 28 21:22:55.701217 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.701199    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0724c864-8d30-4430-b24d-9b9f4b6ea6cc" containerName="istio-proxy"
Jul 28 21:22:55.701288 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.701218    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="0724c864-8d30-4430-b24d-9b9f4b6ea6cc" containerName="istio-proxy"
Jul 28 21:22:55.701288 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.701228    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0724c864-8d30-4430-b24d-9b9f4b6ea6cc" containerName="istio-proxy"
Jul 28 21:22:55.701288 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.701234    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="0724c864-8d30-4430-b24d-9b9f4b6ea6cc" containerName="istio-proxy"
Jul 28 21:22:55.701288 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.701240    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd657d9e-2c36-4ef8-8fba-7c4a506e3e42" containerName="main"
Jul 28 21:22:55.701288 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.701248    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd657d9e-2c36-4ef8-8fba-7c4a506e3e42" containerName="main"
Jul 28 21:22:55.701288 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.701257    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f33b2b1-1345-4691-a967-13b4c6d1964b" containerName="maas-api"
Jul 28 21:22:55.701288 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.701262    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f33b2b1-1345-4691-a967-13b4c6d1964b" containerName="maas-api"
Jul 28 21:22:55.701533 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.701319    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="0724c864-8d30-4430-b24d-9b9f4b6ea6cc" containerName="istio-proxy"
Jul 28 21:22:55.701533 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.701328    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f33b2b1-1345-4691-a967-13b4c6d1964b" containerName="maas-api"
Jul 28 21:22:55.701533 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.701334    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="0724c864-8d30-4430-b24d-9b9f4b6ea6cc" containerName="istio-proxy"
Jul 28 21:22:55.701533 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.701340    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd657d9e-2c36-4ef8-8fba-7c4a506e3e42" containerName="main"
Jul 28 21:22:55.704321 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.704305    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.706840 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.706814    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b16ceb-openshift-default-dockercfg-lpfgf\""
Jul 28 21:22:55.715790 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.715767    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"]
Jul 28 21:22:55.730021 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.729981    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:55.792674 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.792650    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-data\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.792808 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.792702    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-workload-socket\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.792808 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.792728    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-envoy\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.792808 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.792783    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-workload-certs\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.792952 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.792811    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gm9w\" (UniqueName: \"kubernetes.io/projected/64d9bb51-b04d-4249-b694-30e4266c49bb-kube-api-access-9gm9w\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.792952 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.792879    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-credential-socket\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.792952 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.792914    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-podinfo\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.792952 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.792939    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-token\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.793106 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.792953    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d9bb51-b04d-4249-b694-30e4266c49bb-istiod-ca-cert\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.894302 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.894267    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-data\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.894480 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.894352    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-workload-socket\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.894480 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.894389    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-envoy\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.894599 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.894515    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-workload-certs\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.894599 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.894548    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gm9w\" (UniqueName: \"kubernetes.io/projected/64d9bb51-b04d-4249-b694-30e4266c49bb-kube-api-access-9gm9w\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.894599 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.894576    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-credential-socket\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.894751 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.894610    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-podinfo\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.894751 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.894663    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-token\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.894751 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.894715    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-data\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.894955 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.894809    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d9bb51-b04d-4249-b694-30e4266c49bb-istiod-ca-cert\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.894955 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.894844    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-workload-socket\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.894955 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.894945    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-workload-certs\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.895106 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.894969    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-credential-socket\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.895410 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.895388    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d9bb51-b04d-4249-b694-30e4266c49bb-istiod-ca-cert\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.896771 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.896746    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-envoy\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.897000 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.896983    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-podinfo\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.911142 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.911119    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-token\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.911456 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.911437    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gm9w\" (UniqueName: \"kubernetes.io/projected/64d9bb51-b04d-4249-b694-30e4266c49bb-kube-api-access-9gm9w\") pod \"e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") " pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:55.958477 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:55.958402    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:56.015922 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:56.015895    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:56.153027 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:56.152997    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"]
Jul 28 21:22:56.155118 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:22:56.155090    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d9bb51_b04d_4249_b694_30e4266c49bb.slice/crio-fd649b0a891ea6af493910a8a745fbefff4d31d2a89412c93fb4574a67132736 WatchSource:0}: Error finding container fd649b0a891ea6af493910a8a745fbefff4d31d2a89412c93fb4574a67132736: Status 404 returned error can't find the container with id fd649b0a891ea6af493910a8a745fbefff4d31d2a89412c93fb4574a67132736
Jul 28 21:22:56.157191 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:56.157158    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:22:56.157305 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:56.157239    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:22:56.157305 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:56.157284    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:22:56.306796 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:56.306754    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm" event={"ID":"64d9bb51-b04d-4249-b694-30e4266c49bb","Type":"ContainerStarted","Data":"eff6b3ea4d9db904c69c47f7c80c2d2a36aef0d2c8182564dc9b59dae0305955"}
Jul 28 21:22:56.306989 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:56.306816    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm" event={"ID":"64d9bb51-b04d-4249-b694-30e4266c49bb","Type":"ContainerStarted","Data":"fd649b0a891ea6af493910a8a745fbefff4d31d2a89412c93fb4574a67132736"}
Jul 28 21:22:56.330032 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:56.329833    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm" podStartSLOduration=1.3298157609999999 podStartE2EDuration="1.329815761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:22:56.329233995 +0000 UTC m=+2136.946815895" watchObservedRunningTime="2026-07-28 21:22:56.329815761 +0000 UTC m=+2136.947397668"
Jul 28 21:22:57.016178 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:57.016145    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:57.021064 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:57.021032    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:57.314145 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:57.314116    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:57.315128 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:57.315095    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:22:58.222553 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.222518    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:58.321598 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.321562    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:58.467287 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.467252    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"]
Jul 28 21:22:58.470970 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.470947    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"
Jul 28 21:22:58.473756 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.473707    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b16ceb-serving-cert\""
Jul 28 21:22:58.480931 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.480909    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"]
Jul 28 21:22:58.535565 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.535532    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:58.620066 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.620034    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28xcd\" (UniqueName: \"kubernetes.io/projected/a8071aa7-fc30-4043-bde4-edd93b0499dc-kube-api-access-28xcd\") pod \"maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw\" (UID: \"a8071aa7-fc30-4043-bde4-edd93b0499dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"
Jul 28 21:22:58.620217 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.620079    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8071aa7-fc30-4043-bde4-edd93b0499dc-maas-api-tls\") pod \"maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw\" (UID: \"a8071aa7-fc30-4043-bde4-edd93b0499dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"
Jul 28 21:22:58.620217 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.620165    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8071aa7-fc30-4043-bde4-edd93b0499dc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw\" (UID: \"a8071aa7-fc30-4043-bde4-edd93b0499dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"
Jul 28 21:22:58.706023 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.705990    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:58.721035 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.720990    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8071aa7-fc30-4043-bde4-edd93b0499dc-maas-api-tls\") pod \"maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw\" (UID: \"a8071aa7-fc30-4043-bde4-edd93b0499dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"
Jul 28 21:22:58.721154 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.721072    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8071aa7-fc30-4043-bde4-edd93b0499dc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw\" (UID: \"a8071aa7-fc30-4043-bde4-edd93b0499dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"
Jul 28 21:22:58.721154 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.721136    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28xcd\" (UniqueName: \"kubernetes.io/projected/a8071aa7-fc30-4043-bde4-edd93b0499dc-kube-api-access-28xcd\") pod \"maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw\" (UID: \"a8071aa7-fc30-4043-bde4-edd93b0499dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"
Jul 28 21:22:58.721770 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.721752    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8071aa7-fc30-4043-bde4-edd93b0499dc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw\" (UID: \"a8071aa7-fc30-4043-bde4-edd93b0499dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"
Jul 28 21:22:58.723378 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.723361    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8071aa7-fc30-4043-bde4-edd93b0499dc-maas-api-tls\") pod \"maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw\" (UID: \"a8071aa7-fc30-4043-bde4-edd93b0499dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"
Jul 28 21:22:58.729205 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.729154    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28xcd\" (UniqueName: \"kubernetes.io/projected/a8071aa7-fc30-4043-bde4-edd93b0499dc-kube-api-access-28xcd\") pod \"maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw\" (UID: \"a8071aa7-fc30-4043-bde4-edd93b0499dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"
Jul 28 21:22:58.786374 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.786346    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"
Jul 28 21:22:58.921549 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:58.921519    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"]
Jul 28 21:22:58.923937 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:22:58.923906    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8071aa7_fc30_4043_bde4_edd93b0499dc.slice/crio-2ebf0b1166a7090d8e82203e40aa391b1cdfb2c5f695e2beaa59b79f682b6659 WatchSource:0}: Error finding container 2ebf0b1166a7090d8e82203e40aa391b1cdfb2c5f695e2beaa59b79f682b6659: Status 404 returned error can't find the container with id 2ebf0b1166a7090d8e82203e40aa391b1cdfb2c5f695e2beaa59b79f682b6659
Jul 28 21:22:59.068493 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:59.068442    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:59.157658 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:59.157614    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:22:59.326360 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:22:59.326273    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw" event={"ID":"a8071aa7-fc30-4043-bde4-edd93b0499dc","Type":"ContainerStarted","Data":"2ebf0b1166a7090d8e82203e40aa391b1cdfb2c5f695e2beaa59b79f682b6659"}
Jul 28 21:23:00.333400 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:00.333362    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw" event={"ID":"a8071aa7-fc30-4043-bde4-edd93b0499dc","Type":"ContainerStarted","Data":"23a247e93d65c4d10926fbd3df588b775fca7e32651dacd7f76a1623360983d8"}
Jul 28 21:23:00.333922 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:00.333428    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"
Jul 28 21:23:00.348824 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:00.348773    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw" podStartSLOduration=1.854608958 podStartE2EDuration="2.348756373s" totalImagesPullingTime="494.147415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:23:00.348415722 +0000 UTC m=+2140.965997624" watchObservedRunningTime="2026-07-28 21:23:00.348756373 +0000 UTC m=+2140.966338277"
Jul 28 21:23:06.344706 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:06.344671    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"
Jul 28 21:23:12.315549 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:12.315507    2537 patch_prober.go:28] interesting pod/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 28 21:23:12.315981 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:12.315568    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm" podUID="64d9bb51-b04d-4249-b694-30e4266c49bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 28 21:23:12.395786 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:12.395758    2537 generic.go:356] "Generic (PLEG): container finished" podID="64d9bb51-b04d-4249-b694-30e4266c49bb" containerID="eff6b3ea4d9db904c69c47f7c80c2d2a36aef0d2c8182564dc9b59dae0305955" exitCode=0
Jul 28 21:23:12.395952 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:12.395832    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm" event={"ID":"64d9bb51-b04d-4249-b694-30e4266c49bb","Type":"ContainerDied","Data":"eff6b3ea4d9db904c69c47f7c80c2d2a36aef0d2c8182564dc9b59dae0305955"}
Jul 28 21:23:12.396234 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:12.396218    2537 scope.go:117] "RemoveContainer" containerID="eff6b3ea4d9db904c69c47f7c80c2d2a36aef0d2c8182564dc9b59dae0305955"
Jul 28 21:23:12.397206 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:12.397177    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:23:12.397268 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:12.397244    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:23:12.397304 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:12.397277    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:23:13.402841 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:13.402805    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm" event={"ID":"64d9bb51-b04d-4249-b694-30e4266c49bb","Type":"ContainerStarted","Data":"321639d8687844f48d780a6b03aafb90ee9986d32e109e598f19a03dd84924dd"}
Jul 28 21:23:14.021154 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:14.021111    2537 patch_prober.go:28] interesting pod/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 28 21:23:14.021383 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:14.021181    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm" podUID="64d9bb51-b04d-4249-b694-30e4266c49bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 28 21:23:14.026002 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:14.025971    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:23:15.016413 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:15.016377    2537 patch_prober.go:28] interesting pod/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 28 21:23:15.017020 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:15.016989    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm" podUID="64d9bb51-b04d-4249-b694-30e4266c49bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 28 21:23:16.016411 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:16.016375    2537 patch_prober.go:28] interesting pod/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 28 21:23:16.016706 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:16.016675    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm" podUID="64d9bb51-b04d-4249-b694-30e4266c49bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 28 21:23:17.016621 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:17.016575    2537 patch_prober.go:28] interesting pod/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 28 21:23:17.016844 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:17.016641    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm" podUID="64d9bb51-b04d-4249-b694-30e4266c49bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 28 21:23:18.021521 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:18.021491    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:23:18.021908 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:18.021816    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:23:18.022404 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:18.022381    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:23:23.180688 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:23.180652    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:23:23.515212 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:23.515151    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:23:23.821291 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:23.821260    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:23:24.002424 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:24.002388    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:23:26.385522 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:26.385490    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:23:26.480638 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:26.480583    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:23:26.481545 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:26.481524    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"]
Jul 28 21:23:26.493347 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:26.493323    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:23:26.496311 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:26.496269    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-75f962-serving-cert\""
Jul 28 21:23:26.497163 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:26.497141    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"]
Jul 28 21:23:26.551592 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:26.551563    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:23:26.565762 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:26.565734    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d64ebf-0566-4af0-93ba-ca198a05739e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh\" (UID: \"c7d64ebf-0566-4af0-93ba-ca198a05739e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:23:26.565882 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:26.565782    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d64ebf-0566-4af0-93ba-ca198a05739e-maas-api-tls\") pod \"maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh\" (UID: \"c7d64ebf-0566-4af0-93ba-ca198a05739e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:23:26.565882 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:26.565817    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz46x\" (UniqueName: \"kubernetes.io/projected/c7d64ebf-0566-4af0-93ba-ca198a05739e-kube-api-access-jz46x\") pod \"maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh\" (UID: \"c7d64ebf-0566-4af0-93ba-ca198a05739e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:23:26.666894 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:26.666780    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d64ebf-0566-4af0-93ba-ca198a05739e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh\" (UID: \"c7d64ebf-0566-4af0-93ba-ca198a05739e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:23:26.666894 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:26.666846    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d64ebf-0566-4af0-93ba-ca198a05739e-maas-api-tls\") pod \"maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh\" (UID: \"c7d64ebf-0566-4af0-93ba-ca198a05739e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:23:26.667127 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:26.666922    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz46x\" (UniqueName: \"kubernetes.io/projected/c7d64ebf-0566-4af0-93ba-ca198a05739e-kube-api-access-jz46x\") pod \"maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh\" (UID: \"c7d64ebf-0566-4af0-93ba-ca198a05739e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:23:26.667127 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:23:26.667003    2537 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-serving-cert: secret "maas-api-e2e-inf-b-75f962-serving-cert" not found
Jul 28 21:23:26.667127 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:23:26.667084    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d64ebf-0566-4af0-93ba-ca198a05739e-maas-api-tls podName:c7d64ebf-0566-4af0-93ba-ca198a05739e nodeName:}" failed. No retries permitted until 2026-07-28 21:23:27.167061674 +0000 UTC m=+2167.784643568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7d64ebf-0566-4af0-93ba-ca198a05739e-maas-api-tls") pod "maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh" (UID: "c7d64ebf-0566-4af0-93ba-ca198a05739e") : secret "maas-api-e2e-inf-b-75f962-serving-cert" not found
Jul 28 21:23:26.667422 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:26.667402    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d64ebf-0566-4af0-93ba-ca198a05739e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh\" (UID: \"c7d64ebf-0566-4af0-93ba-ca198a05739e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:23:26.676065 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:26.676045    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz46x\" (UniqueName: \"kubernetes.io/projected/c7d64ebf-0566-4af0-93ba-ca198a05739e-kube-api-access-jz46x\") pod \"maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh\" (UID: \"c7d64ebf-0566-4af0-93ba-ca198a05739e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:23:27.011196 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:27.011105    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:23:27.111022 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:27.110985    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:23:27.167932 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:27.167903    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:23:27.171017 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:27.170994    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d64ebf-0566-4af0-93ba-ca198a05739e-maas-api-tls\") pod \"maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh\" (UID: \"c7d64ebf-0566-4af0-93ba-ca198a05739e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:23:27.173443 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:27.173399    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d64ebf-0566-4af0-93ba-ca198a05739e-maas-api-tls\") pod \"maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh\" (UID: \"c7d64ebf-0566-4af0-93ba-ca198a05739e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:23:27.406357 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:27.406320    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:23:27.543722 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:27.543599    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"]
Jul 28 21:23:27.546252 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:23:27.546220    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d64ebf_0566_4af0_93ba_ca198a05739e.slice/crio-cc9a9d79aeb238b8acb5bc82c8d1a4a587744ff247b50e0d2aeb2a4e972aef6e WatchSource:0}: Error finding container cc9a9d79aeb238b8acb5bc82c8d1a4a587744ff247b50e0d2aeb2a4e972aef6e: Status 404 returned error can't find the container with id cc9a9d79aeb238b8acb5bc82c8d1a4a587744ff247b50e0d2aeb2a4e972aef6e
Jul 28 21:23:28.485474 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:28.485436    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh" event={"ID":"c7d64ebf-0566-4af0-93ba-ca198a05739e","Type":"ContainerStarted","Data":"c647f4f569095add90a49df8154594b7300feb4894f23a9870649883a36ced06"}
Jul 28 21:23:28.485935 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:28.485487    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh" event={"ID":"c7d64ebf-0566-4af0-93ba-ca198a05739e","Type":"ContainerStarted","Data":"cc9a9d79aeb238b8acb5bc82c8d1a4a587744ff247b50e0d2aeb2a4e972aef6e"}
Jul 28 21:23:28.485935 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:28.485545    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:23:28.503606 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:28.503557    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh" podStartSLOduration=2.135132038 podStartE2EDuration="2.503545805s" totalImagesPullingTime="368.413767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:23:28.502912694 +0000 UTC m=+2169.120494596" watchObservedRunningTime="2026-07-28 21:23:28.503545805 +0000 UTC m=+2169.121127712"
Jul 28 21:23:34.496463 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:34.496433    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:23:38.203321 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:38.203282    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:23:51.341921 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:51.341878    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:23:55.072220 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:55.072185    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:23:55.123542 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:23:55.123518    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:24:01.671985 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.671950    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"]
Jul 28 21:24:01.675469 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.675451    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.679769 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.679742    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b16ceb\"/\"default-dockercfg-rntbp\""
Jul 28 21:24:01.679920 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.679742    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b16ceb\"/\"test-model-b16ceb-kserve-self-signed-certs\""
Jul 28 21:24:01.679920 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.679744    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b16ceb\"/\"openshift-service-ca.crt\""
Jul 28 21:24:01.679920 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.679748    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b16ceb\"/\"kube-root-ca.crt\""
Jul 28 21:24:01.687153 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.687130    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"]
Jul 28 21:24:01.697328 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.697302    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:24:01.767231 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.767205    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-home\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.767367 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.767248    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-dshm\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.767367 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.767302    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-tmp-dir\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.767367 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.767334    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5ffa42-929b-4539-b4c9-9312a6cb6226-tls-certs\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.767475 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.767412    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-model-cache\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.767475 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.767452    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9zss\" (UniqueName: \"kubernetes.io/projected/1c5ffa42-929b-4539-b4c9-9312a6cb6226-kube-api-access-x9zss\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.868286 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.868255    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-dshm\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.868447 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.868311    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-tmp-dir\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.868447 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.868421    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5ffa42-929b-4539-b4c9-9312a6cb6226-tls-certs\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.868535 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.868496    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-model-cache\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.868593 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.868533    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9zss\" (UniqueName: \"kubernetes.io/projected/1c5ffa42-929b-4539-b4c9-9312a6cb6226-kube-api-access-x9zss\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.868593 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.868588    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-home\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.868699 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.868609    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-tmp-dir\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.868850 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.868832    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-model-cache\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.868960 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.868912    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-home\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.870588 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.870564    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-dshm\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.870912 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.870896    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5ffa42-929b-4539-b4c9-9312a6cb6226-tls-certs\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.877429 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.877410    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9zss\" (UniqueName: \"kubernetes.io/projected/1c5ffa42-929b-4539-b4c9-9312a6cb6226-kube-api-access-x9zss\") pod \"test-model-b16ceb-kserve-584f954c79-26vph\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") " pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:01.984786 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:01.984708    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:02.331665 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:02.331637    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"]
Jul 28 21:24:02.333566 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:24:02.333530    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c5ffa42_929b_4539_b4c9_9312a6cb6226.slice/crio-9865580b21b3b1e9f065f2d193b3436c710bdaaaf6e1296d1801039241943976 WatchSource:0}: Error finding container 9865580b21b3b1e9f065f2d193b3436c710bdaaaf6e1296d1801039241943976: Status 404 returned error can't find the container with id 9865580b21b3b1e9f065f2d193b3436c710bdaaaf6e1296d1801039241943976
Jul 28 21:24:02.657464 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:02.657381    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph" event={"ID":"1c5ffa42-929b-4539-b4c9-9312a6cb6226","Type":"ContainerStarted","Data":"899795677c366bad3dbff9c535f428148b4cca7d3483c98a8a906643bfb957f6"}
Jul 28 21:24:02.657464 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:02.657425    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph" event={"ID":"1c5ffa42-929b-4539-b4c9-9312a6cb6226","Type":"ContainerStarted","Data":"9865580b21b3b1e9f065f2d193b3436c710bdaaaf6e1296d1801039241943976"}
Jul 28 21:24:02.677097 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:02.677043    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph" podStartSLOduration=1.677028377 podStartE2EDuration="1.677028377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:24:02.675239236 +0000 UTC m=+2203.292821140" watchObservedRunningTime="2026-07-28 21:24:02.677028377 +0000 UTC m=+2203.294610278"
Jul 28 21:24:11.985704 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:11.985664    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:11.986200 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:11.985724    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:11.998310 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:11.998281    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:12.721612 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:12.721583    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:24:28.858967 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:24:28.858930    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:21.013828 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:21.013783    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"]
Jul 28 21:25:21.041071 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:21.041037    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:22.209910 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:22.209871    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:32.451956 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:32.451913    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"]
Jul 28 21:25:32.452724 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:32.452685    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw" podUID="a8071aa7-fc30-4043-bde4-edd93b0499dc" containerName="maas-api" containerID="cri-o://23a247e93d65c4d10926fbd3df588b775fca7e32651dacd7f76a1623360983d8" gracePeriod=30
Jul 28 21:25:32.481781 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:32.481748    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:32.542820 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:32.542785    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:32.607870 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:32.607817    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:32.719111 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:32.719004    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:32.916337 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:32.916312    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"
Jul 28 21:25:33.069377 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.069346    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:33.087308 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.087285    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8071aa7-fc30-4043-bde4-edd93b0499dc-maas-api-tls\") pod \"a8071aa7-fc30-4043-bde4-edd93b0499dc\" (UID: \"a8071aa7-fc30-4043-bde4-edd93b0499dc\") "
Jul 28 21:25:33.087409 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.087346    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28xcd\" (UniqueName: \"kubernetes.io/projected/a8071aa7-fc30-4043-bde4-edd93b0499dc-kube-api-access-28xcd\") pod \"a8071aa7-fc30-4043-bde4-edd93b0499dc\" (UID: \"a8071aa7-fc30-4043-bde4-edd93b0499dc\") "
Jul 28 21:25:33.087470 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.087457    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8071aa7-fc30-4043-bde4-edd93b0499dc-openshift-service-ca\") pod \"a8071aa7-fc30-4043-bde4-edd93b0499dc\" (UID: \"a8071aa7-fc30-4043-bde4-edd93b0499dc\") "
Jul 28 21:25:33.087826 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.087797    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8071aa7-fc30-4043-bde4-edd93b0499dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a8071aa7-fc30-4043-bde4-edd93b0499dc" (UID: "a8071aa7-fc30-4043-bde4-edd93b0499dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:25:33.089378 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.089355    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8071aa7-fc30-4043-bde4-edd93b0499dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a8071aa7-fc30-4043-bde4-edd93b0499dc" (UID: "a8071aa7-fc30-4043-bde4-edd93b0499dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:25:33.089462 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.089382    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8071aa7-fc30-4043-bde4-edd93b0499dc-kube-api-access-28xcd" (OuterVolumeSpecName: "kube-api-access-28xcd") pod "a8071aa7-fc30-4043-bde4-edd93b0499dc" (UID: "a8071aa7-fc30-4043-bde4-edd93b0499dc"). InnerVolumeSpecName "kube-api-access-28xcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:25:33.116100 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.116075    2537 generic.go:356] "Generic (PLEG): container finished" podID="a8071aa7-fc30-4043-bde4-edd93b0499dc" containerID="23a247e93d65c4d10926fbd3df588b775fca7e32651dacd7f76a1623360983d8" exitCode=0
Jul 28 21:25:33.116201 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.116115    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"
Jul 28 21:25:33.116201 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.116131    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw" event={"ID":"a8071aa7-fc30-4043-bde4-edd93b0499dc","Type":"ContainerDied","Data":"23a247e93d65c4d10926fbd3df588b775fca7e32651dacd7f76a1623360983d8"}
Jul 28 21:25:33.116201 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.116171    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw" event={"ID":"a8071aa7-fc30-4043-bde4-edd93b0499dc","Type":"ContainerDied","Data":"2ebf0b1166a7090d8e82203e40aa391b1cdfb2c5f695e2beaa59b79f682b6659"}
Jul 28 21:25:33.116201 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.116191    2537 scope.go:117] "RemoveContainer" containerID="23a247e93d65c4d10926fbd3df588b775fca7e32651dacd7f76a1623360983d8"
Jul 28 21:25:33.131671 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.131654    2537 scope.go:117] "RemoveContainer" containerID="23a247e93d65c4d10926fbd3df588b775fca7e32651dacd7f76a1623360983d8"
Jul 28 21:25:33.131949 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:25:33.131926    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a247e93d65c4d10926fbd3df588b775fca7e32651dacd7f76a1623360983d8\": container with ID starting with 23a247e93d65c4d10926fbd3df588b775fca7e32651dacd7f76a1623360983d8 not found: ID does not exist" containerID="23a247e93d65c4d10926fbd3df588b775fca7e32651dacd7f76a1623360983d8"
Jul 28 21:25:33.132000 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.131958    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a247e93d65c4d10926fbd3df588b775fca7e32651dacd7f76a1623360983d8"} err="failed to get container status \"23a247e93d65c4d10926fbd3df588b775fca7e32651dacd7f76a1623360983d8\": rpc error: code = NotFound desc = could not find container \"23a247e93d65c4d10926fbd3df588b775fca7e32651dacd7f76a1623360983d8\": container with ID starting with 23a247e93d65c4d10926fbd3df588b775fca7e32651dacd7f76a1623360983d8 not found: ID does not exist"
Jul 28 21:25:33.146257 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.146224    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"]
Jul 28 21:25:33.150028 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.150006    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b16ceb-77cf588b96-sb8sw"]
Jul 28 21:25:33.188482 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.188456    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28xcd\" (UniqueName: \"kubernetes.io/projected/a8071aa7-fc30-4043-bde4-edd93b0499dc-kube-api-access-28xcd\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:33.188580 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.188483    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8071aa7-fc30-4043-bde4-edd93b0499dc-openshift-service-ca\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:33.188580 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:33.188499    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8071aa7-fc30-4043-bde4-edd93b0499dc-maas-api-tls\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:34.020910 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:34.020878    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8071aa7-fc30-4043-bde4-edd93b0499dc" path="/var/lib/kubelet/pods/a8071aa7-fc30-4043-bde4-edd93b0499dc/volumes"
Jul 28 21:25:36.032867 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.032820    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph" podUID="1c5ffa42-929b-4539-b4c9-9312a6cb6226" containerName="main" containerID="cri-o://899795677c366bad3dbff9c535f428148b4cca7d3483c98a8a906643bfb957f6" gracePeriod=45
Jul 28 21:25:36.283242 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.283178    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:25:36.414850 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.414819    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-home\") pod \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") "
Jul 28 21:25:36.415045 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.414909    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-tmp-dir\") pod \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") "
Jul 28 21:25:36.415045 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.414931    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9zss\" (UniqueName: \"kubernetes.io/projected/1c5ffa42-929b-4539-b4c9-9312a6cb6226-kube-api-access-x9zss\") pod \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") "
Jul 28 21:25:36.415045 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.414968    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5ffa42-929b-4539-b4c9-9312a6cb6226-tls-certs\") pod \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") "
Jul 28 21:25:36.415045 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.414984    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-model-cache\") pod \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") "
Jul 28 21:25:36.415292 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.415081    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-home" (OuterVolumeSpecName: "home") pod "1c5ffa42-929b-4539-b4c9-9312a6cb6226" (UID: "1c5ffa42-929b-4539-b4c9-9312a6cb6226"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:25:36.415292 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.415121    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-dshm\") pod \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\" (UID: \"1c5ffa42-929b-4539-b4c9-9312a6cb6226\") "
Jul 28 21:25:36.415292 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.415228    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c5ffa42-929b-4539-b4c9-9312a6cb6226" (UID: "1c5ffa42-929b-4539-b4c9-9312a6cb6226"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:25:36.415292 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.415239    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c5ffa42-929b-4539-b4c9-9312a6cb6226" (UID: "1c5ffa42-929b-4539-b4c9-9312a6cb6226"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:25:36.415494 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.415424    2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-tmp-dir\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:36.415494 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.415445    2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-model-cache\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:36.415494 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.415458    2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-home\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:36.417236 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.417211    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-dshm" (OuterVolumeSpecName: "dshm") pod "1c5ffa42-929b-4539-b4c9-9312a6cb6226" (UID: "1c5ffa42-929b-4539-b4c9-9312a6cb6226"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:25:36.417346 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.417276    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c5ffa42-929b-4539-b4c9-9312a6cb6226-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c5ffa42-929b-4539-b4c9-9312a6cb6226" (UID: "1c5ffa42-929b-4539-b4c9-9312a6cb6226"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:25:36.417346 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.417291    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5ffa42-929b-4539-b4c9-9312a6cb6226-kube-api-access-x9zss" (OuterVolumeSpecName: "kube-api-access-x9zss") pod "1c5ffa42-929b-4539-b4c9-9312a6cb6226" (UID: "1c5ffa42-929b-4539-b4c9-9312a6cb6226"). InnerVolumeSpecName "kube-api-access-x9zss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:25:36.516188 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.516160    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9zss\" (UniqueName: \"kubernetes.io/projected/1c5ffa42-929b-4539-b4c9-9312a6cb6226-kube-api-access-x9zss\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:36.516188 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.516185    2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5ffa42-929b-4539-b4c9-9312a6cb6226-tls-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:36.516344 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:36.516195    2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5ffa42-929b-4539-b4c9-9312a6cb6226-dshm\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:37.136804 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:37.136769    2537 generic.go:356] "Generic (PLEG): container finished" podID="1c5ffa42-929b-4539-b4c9-9312a6cb6226" containerID="899795677c366bad3dbff9c535f428148b4cca7d3483c98a8a906643bfb957f6" exitCode=0
Jul 28 21:25:37.137253 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:37.136810    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"
Jul 28 21:25:37.137253 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:37.136870    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph" event={"ID":"1c5ffa42-929b-4539-b4c9-9312a6cb6226","Type":"ContainerDied","Data":"899795677c366bad3dbff9c535f428148b4cca7d3483c98a8a906643bfb957f6"}
Jul 28 21:25:37.137253 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:37.136912    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph" event={"ID":"1c5ffa42-929b-4539-b4c9-9312a6cb6226","Type":"ContainerDied","Data":"9865580b21b3b1e9f065f2d193b3436c710bdaaaf6e1296d1801039241943976"}
Jul 28 21:25:37.137253 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:37.136934    2537 scope.go:117] "RemoveContainer" containerID="899795677c366bad3dbff9c535f428148b4cca7d3483c98a8a906643bfb957f6"
Jul 28 21:25:37.146229 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:37.146210    2537 scope.go:117] "RemoveContainer" containerID="899795677c366bad3dbff9c535f428148b4cca7d3483c98a8a906643bfb957f6"
Jul 28 21:25:37.146488 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:25:37.146467    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"899795677c366bad3dbff9c535f428148b4cca7d3483c98a8a906643bfb957f6\": container with ID starting with 899795677c366bad3dbff9c535f428148b4cca7d3483c98a8a906643bfb957f6 not found: ID does not exist" containerID="899795677c366bad3dbff9c535f428148b4cca7d3483c98a8a906643bfb957f6"
Jul 28 21:25:37.146540 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:37.146496    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899795677c366bad3dbff9c535f428148b4cca7d3483c98a8a906643bfb957f6"} err="failed to get container status \"899795677c366bad3dbff9c535f428148b4cca7d3483c98a8a906643bfb957f6\": rpc error: code = NotFound desc = could not find container \"899795677c366bad3dbff9c535f428148b4cca7d3483c98a8a906643bfb957f6\": container with ID starting with 899795677c366bad3dbff9c535f428148b4cca7d3483c98a8a906643bfb957f6 not found: ID does not exist"
Jul 28 21:25:37.159229 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:37.159203    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"]
Jul 28 21:25:37.165550 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:37.165529    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b16ceb/test-model-b16ceb-kserve-584f954c79-26vph"]
Jul 28 21:25:38.021573 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:38.021538    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c5ffa42-929b-4539-b4c9-9312a6cb6226" path="/var/lib/kubelet/pods/1c5ffa42-929b-4539-b4c9-9312a6cb6226/volumes"
Jul 28 21:25:38.894756 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:38.894684    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:38.950925 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:38.950895    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:39.136009 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:39.135979    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"]
Jul 28 21:25:39.136331 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:39.136292    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm" podUID="64d9bb51-b04d-4249-b694-30e4266c49bb" containerName="istio-proxy" containerID="cri-o://321639d8687844f48d780a6b03aafb90ee9986d32e109e598f19a03dd84924dd" gracePeriod=30
Jul 28 21:25:39.157528 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:39.157443    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:39.252890 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:39.252824    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:39.381896 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:39.381840    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:44.393227 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.393195    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:25:44.483761 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.483682    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-token\") pod \"64d9bb51-b04d-4249-b694-30e4266c49bb\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") "
Jul 28 21:25:44.483761 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.483725    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-envoy\") pod \"64d9bb51-b04d-4249-b694-30e4266c49bb\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") "
Jul 28 21:25:44.483761 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.483747    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-data\") pod \"64d9bb51-b04d-4249-b694-30e4266c49bb\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") "
Jul 28 21:25:44.484061 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.483776    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d9bb51-b04d-4249-b694-30e4266c49bb-istiod-ca-cert\") pod \"64d9bb51-b04d-4249-b694-30e4266c49bb\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") "
Jul 28 21:25:44.484061 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.483912    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-podinfo\") pod \"64d9bb51-b04d-4249-b694-30e4266c49bb\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") "
Jul 28 21:25:44.484061 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.483950    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-workload-certs\") pod \"64d9bb51-b04d-4249-b694-30e4266c49bb\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") "
Jul 28 21:25:44.484061 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.484017    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-credential-socket\") pod \"64d9bb51-b04d-4249-b694-30e4266c49bb\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") "
Jul 28 21:25:44.484061 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.484055    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-workload-socket\") pod \"64d9bb51-b04d-4249-b694-30e4266c49bb\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") "
Jul 28 21:25:44.484334 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.484112    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gm9w\" (UniqueName: \"kubernetes.io/projected/64d9bb51-b04d-4249-b694-30e4266c49bb-kube-api-access-9gm9w\") pod \"64d9bb51-b04d-4249-b694-30e4266c49bb\" (UID: \"64d9bb51-b04d-4249-b694-30e4266c49bb\") "
Jul 28 21:25:44.484334 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.484196    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d9bb51-b04d-4249-b694-30e4266c49bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "64d9bb51-b04d-4249-b694-30e4266c49bb" (UID: "64d9bb51-b04d-4249-b694-30e4266c49bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:25:44.484334 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.484220    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "64d9bb51-b04d-4249-b694-30e4266c49bb" (UID: "64d9bb51-b04d-4249-b694-30e4266c49bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:25:44.484503 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.484386    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "64d9bb51-b04d-4249-b694-30e4266c49bb" (UID: "64d9bb51-b04d-4249-b694-30e4266c49bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:25:44.484503 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.484485    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "64d9bb51-b04d-4249-b694-30e4266c49bb" (UID: "64d9bb51-b04d-4249-b694-30e4266c49bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:25:44.484694 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.484671    2537 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d9bb51-b04d-4249-b694-30e4266c49bb-istiod-ca-cert\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:44.484766 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.484698    2537 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-workload-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:44.484766 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.484694    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "64d9bb51-b04d-4249-b694-30e4266c49bb" (UID: "64d9bb51-b04d-4249-b694-30e4266c49bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:25:44.484766 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.484712    2537 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-credential-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:44.486089 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.486064    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "64d9bb51-b04d-4249-b694-30e4266c49bb" (UID: "64d9bb51-b04d-4249-b694-30e4266c49bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:25:44.486089 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.486083    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "64d9bb51-b04d-4249-b694-30e4266c49bb" (UID: "64d9bb51-b04d-4249-b694-30e4266c49bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:25:44.486252 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.486111    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d9bb51-b04d-4249-b694-30e4266c49bb-kube-api-access-9gm9w" (OuterVolumeSpecName: "kube-api-access-9gm9w") pod "64d9bb51-b04d-4249-b694-30e4266c49bb" (UID: "64d9bb51-b04d-4249-b694-30e4266c49bb"). InnerVolumeSpecName "kube-api-access-9gm9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:25:44.486252 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.486219    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "64d9bb51-b04d-4249-b694-30e4266c49bb" (UID: "64d9bb51-b04d-4249-b694-30e4266c49bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 21:25:44.585847 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.585815    2537 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-podinfo\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:44.585847 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.585842    2537 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-workload-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:44.586024 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.585851    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gm9w\" (UniqueName: \"kubernetes.io/projected/64d9bb51-b04d-4249-b694-30e4266c49bb-kube-api-access-9gm9w\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:44.586024 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.585877    2537 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-token\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:44.586024 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.585886    2537 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-envoy\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:44.586024 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:44.585893    2537 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d9bb51-b04d-4249-b694-30e4266c49bb-istio-data\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:45.180342 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:45.180309    2537 generic.go:356] "Generic (PLEG): container finished" podID="64d9bb51-b04d-4249-b694-30e4266c49bb" containerID="321639d8687844f48d780a6b03aafb90ee9986d32e109e598f19a03dd84924dd" exitCode=0
Jul 28 21:25:45.180508 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:45.180353    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"
Jul 28 21:25:45.180508 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:45.180391    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm" event={"ID":"64d9bb51-b04d-4249-b694-30e4266c49bb","Type":"ContainerDied","Data":"321639d8687844f48d780a6b03aafb90ee9986d32e109e598f19a03dd84924dd"}
Jul 28 21:25:45.180508 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:45.180448    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm" event={"ID":"64d9bb51-b04d-4249-b694-30e4266c49bb","Type":"ContainerDied","Data":"fd649b0a891ea6af493910a8a745fbefff4d31d2a89412c93fb4574a67132736"}
Jul 28 21:25:45.180508 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:45.180473    2537 scope.go:117] "RemoveContainer" containerID="321639d8687844f48d780a6b03aafb90ee9986d32e109e598f19a03dd84924dd"
Jul 28 21:25:45.189740 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:45.189722    2537 scope.go:117] "RemoveContainer" containerID="eff6b3ea4d9db904c69c47f7c80c2d2a36aef0d2c8182564dc9b59dae0305955"
Jul 28 21:25:45.199138 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:45.199121    2537 scope.go:117] "RemoveContainer" containerID="321639d8687844f48d780a6b03aafb90ee9986d32e109e598f19a03dd84924dd"
Jul 28 21:25:45.199389 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:25:45.199371    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321639d8687844f48d780a6b03aafb90ee9986d32e109e598f19a03dd84924dd\": container with ID starting with 321639d8687844f48d780a6b03aafb90ee9986d32e109e598f19a03dd84924dd not found: ID does not exist" containerID="321639d8687844f48d780a6b03aafb90ee9986d32e109e598f19a03dd84924dd"
Jul 28 21:25:45.199432 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:45.199398    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321639d8687844f48d780a6b03aafb90ee9986d32e109e598f19a03dd84924dd"} err="failed to get container status \"321639d8687844f48d780a6b03aafb90ee9986d32e109e598f19a03dd84924dd\": rpc error: code = NotFound desc = could not find container \"321639d8687844f48d780a6b03aafb90ee9986d32e109e598f19a03dd84924dd\": container with ID starting with 321639d8687844f48d780a6b03aafb90ee9986d32e109e598f19a03dd84924dd not found: ID does not exist"
Jul 28 21:25:45.199432 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:45.199414    2537 scope.go:117] "RemoveContainer" containerID="eff6b3ea4d9db904c69c47f7c80c2d2a36aef0d2c8182564dc9b59dae0305955"
Jul 28 21:25:45.199646 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:25:45.199632    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff6b3ea4d9db904c69c47f7c80c2d2a36aef0d2c8182564dc9b59dae0305955\": container with ID starting with eff6b3ea4d9db904c69c47f7c80c2d2a36aef0d2c8182564dc9b59dae0305955 not found: ID does not exist" containerID="eff6b3ea4d9db904c69c47f7c80c2d2a36aef0d2c8182564dc9b59dae0305955"
Jul 28 21:25:45.199686 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:45.199649    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff6b3ea4d9db904c69c47f7c80c2d2a36aef0d2c8182564dc9b59dae0305955"} err="failed to get container status \"eff6b3ea4d9db904c69c47f7c80c2d2a36aef0d2c8182564dc9b59dae0305955\": rpc error: code = NotFound desc = could not find container \"eff6b3ea4d9db904c69c47f7c80c2d2a36aef0d2c8182564dc9b59dae0305955\": container with ID starting with eff6b3ea4d9db904c69c47f7c80c2d2a36aef0d2c8182564dc9b59dae0305955 not found: ID does not exist"
Jul 28 21:25:45.205667 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:45.205644    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"]
Jul 28 21:25:45.210877 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:45.210842    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b16ceb-openshift-default-7bdd8bb7d8-5q8pm"]
Jul 28 21:25:46.021326 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:46.021290    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64d9bb51-b04d-4249-b694-30e4266c49bb" path="/var/lib/kubelet/pods/64d9bb51-b04d-4249-b694-30e4266c49bb/volumes"
Jul 28 21:25:50.565659 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:50.565621    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"]
Jul 28 21:25:50.566146 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:50.565894    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh" podUID="c7d64ebf-0566-4af0-93ba-ca198a05739e" containerName="maas-api" containerID="cri-o://c647f4f569095add90a49df8154594b7300feb4894f23a9870649883a36ced06" gracePeriod=30
Jul 28 21:25:50.590773 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:50.590739    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:50.762230 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:50.762192    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:50.857487 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:50.857453    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:50.914089 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:50.914058    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:51.026463 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.026440    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:25:51.138829 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.138747    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d64ebf-0566-4af0-93ba-ca198a05739e-openshift-service-ca\") pod \"c7d64ebf-0566-4af0-93ba-ca198a05739e\" (UID: \"c7d64ebf-0566-4af0-93ba-ca198a05739e\") "
Jul 28 21:25:51.138829 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.138782    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d64ebf-0566-4af0-93ba-ca198a05739e-maas-api-tls\") pod \"c7d64ebf-0566-4af0-93ba-ca198a05739e\" (UID: \"c7d64ebf-0566-4af0-93ba-ca198a05739e\") "
Jul 28 21:25:51.138829 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.138820    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz46x\" (UniqueName: \"kubernetes.io/projected/c7d64ebf-0566-4af0-93ba-ca198a05739e-kube-api-access-jz46x\") pod \"c7d64ebf-0566-4af0-93ba-ca198a05739e\" (UID: \"c7d64ebf-0566-4af0-93ba-ca198a05739e\") "
Jul 28 21:25:51.139218 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.139192    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d64ebf-0566-4af0-93ba-ca198a05739e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7d64ebf-0566-4af0-93ba-ca198a05739e" (UID: "c7d64ebf-0566-4af0-93ba-ca198a05739e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:25:51.140883 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.140849    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d64ebf-0566-4af0-93ba-ca198a05739e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7d64ebf-0566-4af0-93ba-ca198a05739e" (UID: "c7d64ebf-0566-4af0-93ba-ca198a05739e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:25:51.140956 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.140916    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d64ebf-0566-4af0-93ba-ca198a05739e-kube-api-access-jz46x" (OuterVolumeSpecName: "kube-api-access-jz46x") pod "c7d64ebf-0566-4af0-93ba-ca198a05739e" (UID: "c7d64ebf-0566-4af0-93ba-ca198a05739e"). InnerVolumeSpecName "kube-api-access-jz46x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:25:51.210144 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.210099    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:51.211795 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.211768    2537 generic.go:356] "Generic (PLEG): container finished" podID="c7d64ebf-0566-4af0-93ba-ca198a05739e" containerID="c647f4f569095add90a49df8154594b7300feb4894f23a9870649883a36ced06" exitCode=0
Jul 28 21:25:51.211932 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.211810    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"
Jul 28 21:25:51.211932 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.211877    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh" event={"ID":"c7d64ebf-0566-4af0-93ba-ca198a05739e","Type":"ContainerDied","Data":"c647f4f569095add90a49df8154594b7300feb4894f23a9870649883a36ced06"}
Jul 28 21:25:51.212063 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.211959    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh" event={"ID":"c7d64ebf-0566-4af0-93ba-ca198a05739e","Type":"ContainerDied","Data":"cc9a9d79aeb238b8acb5bc82c8d1a4a587744ff247b50e0d2aeb2a4e972aef6e"}
Jul 28 21:25:51.212063 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.211995    2537 scope.go:117] "RemoveContainer" containerID="c647f4f569095add90a49df8154594b7300feb4894f23a9870649883a36ced06"
Jul 28 21:25:51.221720 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.221700    2537 scope.go:117] "RemoveContainer" containerID="c647f4f569095add90a49df8154594b7300feb4894f23a9870649883a36ced06"
Jul 28 21:25:51.221978 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:25:51.221956    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c647f4f569095add90a49df8154594b7300feb4894f23a9870649883a36ced06\": container with ID starting with c647f4f569095add90a49df8154594b7300feb4894f23a9870649883a36ced06 not found: ID does not exist" containerID="c647f4f569095add90a49df8154594b7300feb4894f23a9870649883a36ced06"
Jul 28 21:25:51.222059 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.221984    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c647f4f569095add90a49df8154594b7300feb4894f23a9870649883a36ced06"} err="failed to get container status \"c647f4f569095add90a49df8154594b7300feb4894f23a9870649883a36ced06\": rpc error: code = NotFound desc = could not find container \"c647f4f569095add90a49df8154594b7300feb4894f23a9870649883a36ced06\": container with ID starting with c647f4f569095add90a49df8154594b7300feb4894f23a9870649883a36ced06 not found: ID does not exist"
Jul 28 21:25:51.235444 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.235425    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"]
Jul 28 21:25:51.237257 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.237235    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75f962-6fb575c5f9-ps5xh"]
Jul 28 21:25:51.240319 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.240291    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d64ebf-0566-4af0-93ba-ca198a05739e-openshift-service-ca\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:51.240424 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.240322    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d64ebf-0566-4af0-93ba-ca198a05739e-maas-api-tls\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:51.240424 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:51.240338    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz46x\" (UniqueName: \"kubernetes.io/projected/c7d64ebf-0566-4af0-93ba-ca198a05739e-kube-api-access-jz46x\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:25:52.021112 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:52.021053    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d64ebf-0566-4af0-93ba-ca198a05739e" path="/var/lib/kubelet/pods/c7d64ebf-0566-4af0-93ba-ca198a05739e/volumes"
Jul 28 21:25:56.900717 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:56.900656    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:57.000995 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.000962    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:57.158540 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.158453    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:57.375411 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.375363    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:57.710428 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.710386    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:57.988375 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.988287    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"]
Jul 28 21:25:57.988822 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.988683    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d64ebf-0566-4af0-93ba-ca198a05739e" containerName="maas-api"
Jul 28 21:25:57.988822 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.988695    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d64ebf-0566-4af0-93ba-ca198a05739e" containerName="maas-api"
Jul 28 21:25:57.988822 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.988711    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64d9bb51-b04d-4249-b694-30e4266c49bb" containerName="istio-proxy"
Jul 28 21:25:57.988822 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.988717    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="64d9bb51-b04d-4249-b694-30e4266c49bb" containerName="istio-proxy"
Jul 28 21:25:57.988822 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.988732    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c5ffa42-929b-4539-b4c9-9312a6cb6226" containerName="main"
Jul 28 21:25:57.988822 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.988741    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c5ffa42-929b-4539-b4c9-9312a6cb6226" containerName="main"
Jul 28 21:25:57.988822 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.988751    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64d9bb51-b04d-4249-b694-30e4266c49bb" containerName="istio-proxy"
Jul 28 21:25:57.988822 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.988756    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="64d9bb51-b04d-4249-b694-30e4266c49bb" containerName="istio-proxy"
Jul 28 21:25:57.988822 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.988762    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8071aa7-fc30-4043-bde4-edd93b0499dc" containerName="maas-api"
Jul 28 21:25:57.988822 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.988767    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8071aa7-fc30-4043-bde4-edd93b0499dc" containerName="maas-api"
Jul 28 21:25:57.988822 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.988826    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="64d9bb51-b04d-4249-b694-30e4266c49bb" containerName="istio-proxy"
Jul 28 21:25:57.989453 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.988835    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d64ebf-0566-4af0-93ba-ca198a05739e" containerName="maas-api"
Jul 28 21:25:57.989453 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.988842    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8071aa7-fc30-4043-bde4-edd93b0499dc" containerName="maas-api"
Jul 28 21:25:57.989453 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.988850    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c5ffa42-929b-4539-b4c9-9312a6cb6226" containerName="main"
Jul 28 21:25:57.991962 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.991942    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:57.994733 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:57.994713    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3c4e74-openshift-default-dockercfg-f4vr7\""
Jul 28 21:25:58.002567 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.002545    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"]
Jul 28 21:25:58.014127 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.014102    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:58.097628 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.097582    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-credential-socket\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.097628 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.097643    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-envoy\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.097897 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.097665    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-workload-socket\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.097897 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.097717    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-token\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.097897 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.097748    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-podinfo\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.097897 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.097784    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-data\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.097897 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.097804    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7cnk\" (UniqueName: \"kubernetes.io/projected/0644cd8f-b07b-4c7a-b763-01b54daf7c03-kube-api-access-g7cnk\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.097897 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.097845    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istiod-ca-cert\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.098197 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.097970    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-workload-certs\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.199212 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.199184    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-token\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.199381 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.199217    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-podinfo\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.199381 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.199243    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-data\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.199381 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.199261    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7cnk\" (UniqueName: \"kubernetes.io/projected/0644cd8f-b07b-4c7a-b763-01b54daf7c03-kube-api-access-g7cnk\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.199381 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.199302    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istiod-ca-cert\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.199381 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.199348    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-workload-certs\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.199652 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.199384    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-credential-socket\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.199652 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.199413    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-envoy\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.199652 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.199441    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-workload-socket\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.199652 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.199610    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-data\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.199897 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.199709    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-workload-certs\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.199897 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.199785    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-credential-socket\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.199897 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.199822    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-workload-socket\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.200358 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.200333    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istiod-ca-cert\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.201497 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.201473    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-envoy\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.201776 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.201759    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-podinfo\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.210590 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.210549    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-token\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.210590 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.210554    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7cnk\" (UniqueName: \"kubernetes.io/projected/0644cd8f-b07b-4c7a-b763-01b54daf7c03-kube-api-access-g7cnk\") pod \"e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") " pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.291653 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.291564    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:25:58.302915 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.302885    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:58.441227 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.441199    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"]
Jul 28 21:25:58.442973 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:25:58.442946    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0644cd8f_b07b_4c7a_b763_01b54daf7c03.slice/crio-65899503b0858a59b0077f5cfa4a8ec3a38fb7876e5985da7a890ec14c9faaa4 WatchSource:0}: Error finding container 65899503b0858a59b0077f5cfa4a8ec3a38fb7876e5985da7a890ec14c9faaa4: Status 404 returned error can't find the container with id 65899503b0858a59b0077f5cfa4a8ec3a38fb7876e5985da7a890ec14c9faaa4
Jul 28 21:25:58.445165 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.445136    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:25:58.445242 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.445201    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:25:58.445242 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:58.445232    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:25:59.253664 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:59.253627    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd" event={"ID":"0644cd8f-b07b-4c7a-b763-01b54daf7c03","Type":"ContainerStarted","Data":"d51cc424c298f549b5d6a0afed3760d637f4db2342e92d7eff05e288a20d9b80"}
Jul 28 21:25:59.254069 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:59.253678    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd" event={"ID":"0644cd8f-b07b-4c7a-b763-01b54daf7c03","Type":"ContainerStarted","Data":"65899503b0858a59b0077f5cfa4a8ec3a38fb7876e5985da7a890ec14c9faaa4"}
Jul 28 21:25:59.282185 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:59.282135    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd" podStartSLOduration=2.282115611 podStartE2EDuration="2.282115611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:25:59.281903482 +0000 UTC m=+2319.899485464" watchObservedRunningTime="2026-07-28 21:25:59.282115611 +0000 UTC m=+2319.899697513"
Jul 28 21:25:59.303487 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:59.303463    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:25:59.308492 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:25:59.308471    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:26:00.259114 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:00.259086    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:26:00.260118 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:00.260095    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:26:01.169911 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.169841    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:01.256386 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.256350    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"]
Jul 28 21:26:01.256953 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.256935    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="64d9bb51-b04d-4249-b694-30e4266c49bb" containerName="istio-proxy"
Jul 28 21:26:01.260096 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.260075    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"
Jul 28 21:26:01.263109 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.263089    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3c4e74-serving-cert\""
Jul 28 21:26:01.268945 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.268907    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"]
Jul 28 21:26:01.293782 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.293749    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:01.329164 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.329134    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cc8e20-5802-4534-b231-4c26f3aabc91-maas-api-tls\") pod \"maas-api-e2e-api-a-3c4e74-57bb974658-b9svb\" (UID: \"04cc8e20-5802-4534-b231-4c26f3aabc91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"
Jul 28 21:26:01.329350 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.329324    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgml6\" (UniqueName: \"kubernetes.io/projected/04cc8e20-5802-4534-b231-4c26f3aabc91-kube-api-access-zgml6\") pod \"maas-api-e2e-api-a-3c4e74-57bb974658-b9svb\" (UID: \"04cc8e20-5802-4534-b231-4c26f3aabc91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"
Jul 28 21:26:01.329430 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.329408    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cc8e20-5802-4534-b231-4c26f3aabc91-openshift-service-ca\") pod \"maas-api-e2e-api-a-3c4e74-57bb974658-b9svb\" (UID: \"04cc8e20-5802-4534-b231-4c26f3aabc91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"
Jul 28 21:26:01.389007 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.388972    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:01.430347 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.430255    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgml6\" (UniqueName: \"kubernetes.io/projected/04cc8e20-5802-4534-b231-4c26f3aabc91-kube-api-access-zgml6\") pod \"maas-api-e2e-api-a-3c4e74-57bb974658-b9svb\" (UID: \"04cc8e20-5802-4534-b231-4c26f3aabc91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"
Jul 28 21:26:01.430347 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.430304    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cc8e20-5802-4534-b231-4c26f3aabc91-openshift-service-ca\") pod \"maas-api-e2e-api-a-3c4e74-57bb974658-b9svb\" (UID: \"04cc8e20-5802-4534-b231-4c26f3aabc91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"
Jul 28 21:26:01.430591 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.430368    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cc8e20-5802-4534-b231-4c26f3aabc91-maas-api-tls\") pod \"maas-api-e2e-api-a-3c4e74-57bb974658-b9svb\" (UID: \"04cc8e20-5802-4534-b231-4c26f3aabc91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"
Jul 28 21:26:01.430990 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.430966    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cc8e20-5802-4534-b231-4c26f3aabc91-openshift-service-ca\") pod \"maas-api-e2e-api-a-3c4e74-57bb974658-b9svb\" (UID: \"04cc8e20-5802-4534-b231-4c26f3aabc91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"
Jul 28 21:26:01.433027 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.432998    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cc8e20-5802-4534-b231-4c26f3aabc91-maas-api-tls\") pod \"maas-api-e2e-api-a-3c4e74-57bb974658-b9svb\" (UID: \"04cc8e20-5802-4534-b231-4c26f3aabc91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"
Jul 28 21:26:01.441680 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.441661    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgml6\" (UniqueName: \"kubernetes.io/projected/04cc8e20-5802-4534-b231-4c26f3aabc91-kube-api-access-zgml6\") pod \"maas-api-e2e-api-a-3c4e74-57bb974658-b9svb\" (UID: \"04cc8e20-5802-4534-b231-4c26f3aabc91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"
Jul 28 21:26:01.488470 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.488445    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:01.574242 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.574216    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"
Jul 28 21:26:01.712179 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.712109    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"]
Jul 28 21:26:01.714510 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:26:01.714477    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cc8e20_5802_4534_b231_4c26f3aabc91.slice/crio-1a3ff409665ddb87538d9ce5c1c5d45564d63b8626efe46a1a0ae916abe5b689 WatchSource:0}: Error finding container 1a3ff409665ddb87538d9ce5c1c5d45564d63b8626efe46a1a0ae916abe5b689: Status 404 returned error can't find the container with id 1a3ff409665ddb87538d9ce5c1c5d45564d63b8626efe46a1a0ae916abe5b689
Jul 28 21:26:01.715879 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.715838    2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 21:26:01.943268 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:01.943235    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:02.044261 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:02.044167    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:02.270829 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:02.270792    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb" event={"ID":"04cc8e20-5802-4534-b231-4c26f3aabc91","Type":"ContainerStarted","Data":"1a3ff409665ddb87538d9ce5c1c5d45564d63b8626efe46a1a0ae916abe5b689"}
Jul 28 21:26:03.277598 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:03.277560    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb" event={"ID":"04cc8e20-5802-4534-b231-4c26f3aabc91","Type":"ContainerStarted","Data":"a3149b082837d7e2f31dd31ae168b4d9a296ee373a233ff031f35788fdd47ab2"}
Jul 28 21:26:03.278067 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:03.277635    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"
Jul 28 21:26:03.297942 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:03.297882    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb" podStartSLOduration=1.738057853 podStartE2EDuration="2.297841978s" totalImagesPullingTime="559.784125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:26:03.296476658 +0000 UTC m=+2323.914058572" watchObservedRunningTime="2026-07-28 21:26:03.297841978 +0000 UTC m=+2323.915423881"
Jul 28 21:26:09.288926 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:09.288887    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"
Jul 28 21:26:12.803102 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:12.803058    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:15.566141 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:15.564849    2537 patch_prober.go:28] interesting pod/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": read tcp 10.133.0.2:60376->10.133.0.68:15021: read: connection reset by peer" start-of-body=
Jul 28 21:26:15.566141 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:15.564969    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd" podUID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": read tcp 10.133.0.2:60376->10.133.0.68:15021: read: connection reset by peer"
Jul 28 21:26:16.353213 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:16.353181    2537 generic.go:356] "Generic (PLEG): container finished" podID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" containerID="d51cc424c298f549b5d6a0afed3760d637f4db2342e92d7eff05e288a20d9b80" exitCode=0
Jul 28 21:26:16.353393 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:16.353265    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd" event={"ID":"0644cd8f-b07b-4c7a-b763-01b54daf7c03","Type":"ContainerDied","Data":"d51cc424c298f549b5d6a0afed3760d637f4db2342e92d7eff05e288a20d9b80"}
Jul 28 21:26:16.353648 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:16.353632    2537 scope.go:117] "RemoveContainer" containerID="d51cc424c298f549b5d6a0afed3760d637f4db2342e92d7eff05e288a20d9b80"
Jul 28 21:26:16.354578 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:16.354550    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:26:16.354634 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:16.354617    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:26:16.354675 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:16.354649    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:26:17.359933 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:17.359846    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd" event={"ID":"0644cd8f-b07b-4c7a-b763-01b54daf7c03","Type":"ContainerStarted","Data":"58342f7db82f5ada33850f0362ac4049495575684fd299eab93dd1fe5ea6b380"}
Jul 28 21:26:18.303744 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:18.303709    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:26:18.303955 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:18.303906    2537 patch_prober.go:28] interesting pod/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 28 21:26:18.304009 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:18.303976    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd" podUID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 28 21:26:19.304205 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:19.304165    2537 patch_prober.go:28] interesting pod/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 28 21:26:19.304590 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:19.304222    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd" podUID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 28 21:26:20.303473 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:20.303436    2537 patch_prober.go:28] interesting pod/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 28 21:26:20.303638 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:20.303516    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd" podUID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 28 21:26:21.303694 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:21.303658    2537 patch_prober.go:28] interesting pod/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 28 21:26:21.304141 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:21.303715    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd" podUID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 28 21:26:22.307530 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:22.307495    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:26:22.307974 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:22.307783    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:26:22.308458 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:22.308439    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:26:26.130695 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:26.130655    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:26.599584 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:26.599546    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:26.894494 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:26.894395    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:27.193888 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:27.193789    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:30.269266 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:30.269225    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:30.369406 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:30.369371    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:30.468217 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:30.468173    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:30.846981 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:30.846930    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:31.082164 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:31.082135    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:31.185823 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:31.185737    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:26:55.218864 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:26:55.218789    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:02.609212 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:02.609175    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:13.674044 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:13.674009    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"]
Jul 28 21:27:13.674875 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:13.674803    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb" podUID="04cc8e20-5802-4534-b231-4c26f3aabc91" containerName="maas-api" containerID="cri-o://a3149b082837d7e2f31dd31ae168b4d9a296ee373a233ff031f35788fdd47ab2" gracePeriod=30
Jul 28 21:27:13.700258 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:13.700229    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:13.798923 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:13.798887    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:13.871032 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:13.870997    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:13.932883 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:13.932787    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:13.939424 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:13.939405    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"
Jul 28 21:27:13.998810 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:13.998782    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgml6\" (UniqueName: \"kubernetes.io/projected/04cc8e20-5802-4534-b231-4c26f3aabc91-kube-api-access-zgml6\") pod \"04cc8e20-5802-4534-b231-4c26f3aabc91\" (UID: \"04cc8e20-5802-4534-b231-4c26f3aabc91\") "
Jul 28 21:27:13.998983 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:13.998917    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cc8e20-5802-4534-b231-4c26f3aabc91-openshift-service-ca\") pod \"04cc8e20-5802-4534-b231-4c26f3aabc91\" (UID: \"04cc8e20-5802-4534-b231-4c26f3aabc91\") "
Jul 28 21:27:13.999251 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:13.999220    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04cc8e20-5802-4534-b231-4c26f3aabc91-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "04cc8e20-5802-4534-b231-4c26f3aabc91" (UID: "04cc8e20-5802-4534-b231-4c26f3aabc91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:27:14.000851 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.000829    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04cc8e20-5802-4534-b231-4c26f3aabc91-kube-api-access-zgml6" (OuterVolumeSpecName: "kube-api-access-zgml6") pod "04cc8e20-5802-4534-b231-4c26f3aabc91" (UID: "04cc8e20-5802-4534-b231-4c26f3aabc91"). InnerVolumeSpecName "kube-api-access-zgml6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:27:14.099911 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.099877    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cc8e20-5802-4534-b231-4c26f3aabc91-maas-api-tls\") pod \"04cc8e20-5802-4534-b231-4c26f3aabc91\" (UID: \"04cc8e20-5802-4534-b231-4c26f3aabc91\") "
Jul 28 21:27:14.100213 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.100173    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgml6\" (UniqueName: \"kubernetes.io/projected/04cc8e20-5802-4534-b231-4c26f3aabc91-kube-api-access-zgml6\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:27:14.100213 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.100215    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cc8e20-5802-4534-b231-4c26f3aabc91-openshift-service-ca\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:27:14.101909 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.101877    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04cc8e20-5802-4534-b231-4c26f3aabc91-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "04cc8e20-5802-4534-b231-4c26f3aabc91" (UID: "04cc8e20-5802-4534-b231-4c26f3aabc91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:27:14.201541 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.201475    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cc8e20-5802-4534-b231-4c26f3aabc91-maas-api-tls\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:27:14.230304 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.230270    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:14.649040 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.649006    2537 generic.go:356] "Generic (PLEG): container finished" podID="04cc8e20-5802-4534-b231-4c26f3aabc91" containerID="a3149b082837d7e2f31dd31ae168b4d9a296ee373a233ff031f35788fdd47ab2" exitCode=0
Jul 28 21:27:14.649225 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.649053    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"
Jul 28 21:27:14.649225 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.649089    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb" event={"ID":"04cc8e20-5802-4534-b231-4c26f3aabc91","Type":"ContainerDied","Data":"a3149b082837d7e2f31dd31ae168b4d9a296ee373a233ff031f35788fdd47ab2"}
Jul 28 21:27:14.649225 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.649146    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb" event={"ID":"04cc8e20-5802-4534-b231-4c26f3aabc91","Type":"ContainerDied","Data":"1a3ff409665ddb87538d9ce5c1c5d45564d63b8626efe46a1a0ae916abe5b689"}
Jul 28 21:27:14.649225 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.649168    2537 scope.go:117] "RemoveContainer" containerID="a3149b082837d7e2f31dd31ae168b4d9a296ee373a233ff031f35788fdd47ab2"
Jul 28 21:27:14.658244 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.658224    2537 scope.go:117] "RemoveContainer" containerID="a3149b082837d7e2f31dd31ae168b4d9a296ee373a233ff031f35788fdd47ab2"
Jul 28 21:27:14.658478 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:27:14.658460    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3149b082837d7e2f31dd31ae168b4d9a296ee373a233ff031f35788fdd47ab2\": container with ID starting with a3149b082837d7e2f31dd31ae168b4d9a296ee373a233ff031f35788fdd47ab2 not found: ID does not exist" containerID="a3149b082837d7e2f31dd31ae168b4d9a296ee373a233ff031f35788fdd47ab2"
Jul 28 21:27:14.658536 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.658484    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3149b082837d7e2f31dd31ae168b4d9a296ee373a233ff031f35788fdd47ab2"} err="failed to get container status \"a3149b082837d7e2f31dd31ae168b4d9a296ee373a233ff031f35788fdd47ab2\": rpc error: code = NotFound desc = could not find container \"a3149b082837d7e2f31dd31ae168b4d9a296ee373a233ff031f35788fdd47ab2\": container with ID starting with a3149b082837d7e2f31dd31ae168b4d9a296ee373a233ff031f35788fdd47ab2 not found: ID does not exist"
Jul 28 21:27:14.670719 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.670697    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"]
Jul 28 21:27:14.676098 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:14.676077    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c4e74-57bb974658-b9svb"]
Jul 28 21:27:16.020706 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:16.020671    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04cc8e20-5802-4534-b231-4c26f3aabc91" path="/var/lib/kubelet/pods/04cc8e20-5802-4534-b231-4c26f3aabc91/volumes"
Jul 28 21:27:20.035826 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:20.035796    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:27:20.041414 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:20.041394    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:27:20.143590 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:20.143553    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:20.198868 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:20.198820    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:20.367705 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:20.367671    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"]
Jul 28 21:27:20.367977 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:20.367947    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd" podUID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" containerName="istio-proxy" containerID="cri-o://58342f7db82f5ada33850f0362ac4049495575684fd299eab93dd1fe5ea6b380" gracePeriod=30
Jul 28 21:27:20.384818 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:20.384794    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:20.595195 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:20.595162    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:22.310215 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:22.310175    2537 patch_prober.go:28] interesting pod/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 21:27:22.310578 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:22.310234    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd" podUID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 21:27:25.632966 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.632909    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:27:25.700121 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.700092    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7cnk\" (UniqueName: \"kubernetes.io/projected/0644cd8f-b07b-4c7a-b763-01b54daf7c03-kube-api-access-g7cnk\") pod \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") "
Jul 28 21:27:25.700277 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.700136    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-token\") pod \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") "
Jul 28 21:27:25.700277 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.700173    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-workload-socket\") pod \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") "
Jul 28 21:27:25.700395 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.700290    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-data\") pod \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") "
Jul 28 21:27:25.700395 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.700341    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istiod-ca-cert\") pod \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") "
Jul 28 21:27:25.700500 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.700418    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-podinfo\") pod \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") "
Jul 28 21:27:25.700500 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.700438    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0644cd8f-b07b-4c7a-b763-01b54daf7c03" (UID: "0644cd8f-b07b-4c7a-b763-01b54daf7c03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:27:25.700500 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.700444    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-workload-certs\") pod \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") "
Jul 28 21:27:25.700658 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.700524    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-envoy\") pod \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") "
Jul 28 21:27:25.700658 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.700597    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-credential-socket\") pod \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\" (UID: \"0644cd8f-b07b-4c7a-b763-01b54daf7c03\") "
Jul 28 21:27:25.700771 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.700652    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0644cd8f-b07b-4c7a-b763-01b54daf7c03" (UID: "0644cd8f-b07b-4c7a-b763-01b54daf7c03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:27:25.700771 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.700713    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0644cd8f-b07b-4c7a-b763-01b54daf7c03" (UID: "0644cd8f-b07b-4c7a-b763-01b54daf7c03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:27:25.700935 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.700907    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0644cd8f-b07b-4c7a-b763-01b54daf7c03" (UID: "0644cd8f-b07b-4c7a-b763-01b54daf7c03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:27:25.701270 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.701221    2537 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istiod-ca-cert\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:27:25.701270 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.701247    2537 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-workload-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:27:25.701270 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.701263    2537 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-credential-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:27:25.701497 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.701279    2537 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-workload-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:27:25.701576 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.701544    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-data" (OuterVolumeSpecName: "istio-data") pod "0644cd8f-b07b-4c7a-b763-01b54daf7c03" (UID: "0644cd8f-b07b-4c7a-b763-01b54daf7c03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:27:25.702730 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.702689    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0644cd8f-b07b-4c7a-b763-01b54daf7c03-kube-api-access-g7cnk" (OuterVolumeSpecName: "kube-api-access-g7cnk") pod "0644cd8f-b07b-4c7a-b763-01b54daf7c03" (UID: "0644cd8f-b07b-4c7a-b763-01b54daf7c03"). InnerVolumeSpecName "kube-api-access-g7cnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:27:25.702829 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.702746    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0644cd8f-b07b-4c7a-b763-01b54daf7c03" (UID: "0644cd8f-b07b-4c7a-b763-01b54daf7c03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 21:27:25.702829 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.702768    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0644cd8f-b07b-4c7a-b763-01b54daf7c03" (UID: "0644cd8f-b07b-4c7a-b763-01b54daf7c03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:27:25.702829 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.702793    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-token" (OuterVolumeSpecName: "istio-token") pod "0644cd8f-b07b-4c7a-b763-01b54daf7c03" (UID: "0644cd8f-b07b-4c7a-b763-01b54daf7c03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:27:25.709541 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.709512    2537 generic.go:356] "Generic (PLEG): container finished" podID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" containerID="58342f7db82f5ada33850f0362ac4049495575684fd299eab93dd1fe5ea6b380" exitCode=0
Jul 28 21:27:25.709642 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.709576    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"
Jul 28 21:27:25.709689 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.709575    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd" event={"ID":"0644cd8f-b07b-4c7a-b763-01b54daf7c03","Type":"ContainerDied","Data":"58342f7db82f5ada33850f0362ac4049495575684fd299eab93dd1fe5ea6b380"}
Jul 28 21:27:25.709733 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.709695    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd" event={"ID":"0644cd8f-b07b-4c7a-b763-01b54daf7c03","Type":"ContainerDied","Data":"65899503b0858a59b0077f5cfa4a8ec3a38fb7876e5985da7a890ec14c9faaa4"}
Jul 28 21:27:25.709733 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.709728    2537 scope.go:117] "RemoveContainer" containerID="58342f7db82f5ada33850f0362ac4049495575684fd299eab93dd1fe5ea6b380"
Jul 28 21:27:25.723087 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.723066    2537 scope.go:117] "RemoveContainer" containerID="d51cc424c298f549b5d6a0afed3760d637f4db2342e92d7eff05e288a20d9b80"
Jul 28 21:27:25.730591 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.730573    2537 scope.go:117] "RemoveContainer" containerID="58342f7db82f5ada33850f0362ac4049495575684fd299eab93dd1fe5ea6b380"
Jul 28 21:27:25.730915 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:27:25.730894    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58342f7db82f5ada33850f0362ac4049495575684fd299eab93dd1fe5ea6b380\": container with ID starting with 58342f7db82f5ada33850f0362ac4049495575684fd299eab93dd1fe5ea6b380 not found: ID does not exist" containerID="58342f7db82f5ada33850f0362ac4049495575684fd299eab93dd1fe5ea6b380"
Jul 28 21:27:25.731008 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.730924    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58342f7db82f5ada33850f0362ac4049495575684fd299eab93dd1fe5ea6b380"} err="failed to get container status \"58342f7db82f5ada33850f0362ac4049495575684fd299eab93dd1fe5ea6b380\": rpc error: code = NotFound desc = could not find container \"58342f7db82f5ada33850f0362ac4049495575684fd299eab93dd1fe5ea6b380\": container with ID starting with 58342f7db82f5ada33850f0362ac4049495575684fd299eab93dd1fe5ea6b380 not found: ID does not exist"
Jul 28 21:27:25.731008 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.730945    2537 scope.go:117] "RemoveContainer" containerID="d51cc424c298f549b5d6a0afed3760d637f4db2342e92d7eff05e288a20d9b80"
Jul 28 21:27:25.731262 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:27:25.731241    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51cc424c298f549b5d6a0afed3760d637f4db2342e92d7eff05e288a20d9b80\": container with ID starting with d51cc424c298f549b5d6a0afed3760d637f4db2342e92d7eff05e288a20d9b80 not found: ID does not exist" containerID="d51cc424c298f549b5d6a0afed3760d637f4db2342e92d7eff05e288a20d9b80"
Jul 28 21:27:25.731355 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.731268    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51cc424c298f549b5d6a0afed3760d637f4db2342e92d7eff05e288a20d9b80"} err="failed to get container status \"d51cc424c298f549b5d6a0afed3760d637f4db2342e92d7eff05e288a20d9b80\": rpc error: code = NotFound desc = could not find container \"d51cc424c298f549b5d6a0afed3760d637f4db2342e92d7eff05e288a20d9b80\": container with ID starting with d51cc424c298f549b5d6a0afed3760d637f4db2342e92d7eff05e288a20d9b80 not found: ID does not exist"
Jul 28 21:27:25.733741 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.733673    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"]
Jul 28 21:27:25.736060 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.736040    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3c4e74-openshift-default-556c599d4-57bdd"]
Jul 28 21:27:25.801944 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.801923    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7cnk\" (UniqueName: \"kubernetes.io/projected/0644cd8f-b07b-4c7a-b763-01b54daf7c03-kube-api-access-g7cnk\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:27:25.801944 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.801944    2537 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-token\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:27:25.802211 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.801954    2537 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-data\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:27:25.802211 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.801962    2537 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-podinfo\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:27:25.802211 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:25.801970    2537 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0644cd8f-b07b-4c7a-b763-01b54daf7c03-istio-envoy\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:27:26.021399 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:26.021327    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" path="/var/lib/kubelet/pods/0644cd8f-b07b-4c7a-b763-01b54daf7c03/volumes"
Jul 28 21:27:30.838690 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:30.838655    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:30.910560 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:30.910519    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:30.969840 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:30.969807    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:31.045873 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:31.045825    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:31.385611 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:31.385576    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:37.207703 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:37.207666    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:37.297232 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:37.297197    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:37.473732 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:37.473623    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:37.692451 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:37.692408    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:38.010052 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.010013    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:38.291366 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.291277    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"]
Jul 28 21:27:38.291888 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.291839    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" containerName="istio-proxy"
Jul 28 21:27:38.291888 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.291876    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" containerName="istio-proxy"
Jul 28 21:27:38.292038 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.291924    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04cc8e20-5802-4534-b231-4c26f3aabc91" containerName="maas-api"
Jul 28 21:27:38.292038 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.291934    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="04cc8e20-5802-4534-b231-4c26f3aabc91" containerName="maas-api"
Jul 28 21:27:38.292038 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.292036    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" containerName="istio-proxy"
Jul 28 21:27:38.292200 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.292050    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" containerName="istio-proxy"
Jul 28 21:27:38.292200 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.292064    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="04cc8e20-5802-4534-b231-4c26f3aabc91" containerName="maas-api"
Jul 28 21:27:38.295264 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.295243    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.298024 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.298002    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-1096ca-openshift-default-dockercfg-4ljbb\""
Jul 28 21:27:38.306268 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.306245    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"]
Jul 28 21:27:38.308436 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.308357    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:38.407821 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.407772    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-workload-certs\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.408000 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.407925    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-envoy\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.408000 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.407949    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-podinfo\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.408000 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.407971    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-token\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.408193 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.408005    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-data\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.408193 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.408083    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-credential-socket\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.408193 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.408142    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbxzf\" (UniqueName: \"kubernetes.io/projected/d51b6769-c17d-4e9e-85f8-fd0e819273f8-kube-api-access-cbxzf\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.408292 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.408199    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istiod-ca-cert\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.408292 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.408228    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-workload-socket\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.509417 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.509381    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-envoy\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.509417 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.509418    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-podinfo\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.509655 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.509445    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-token\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.509655 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.509478    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-data\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.509655 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.509510    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-credential-socket\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.509655 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.509557    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbxzf\" (UniqueName: \"kubernetes.io/projected/d51b6769-c17d-4e9e-85f8-fd0e819273f8-kube-api-access-cbxzf\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.509655 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.509601    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istiod-ca-cert\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.509655 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.509631    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-workload-socket\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.510022 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.509666    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-workload-certs\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.510022 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.509920    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-data\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.510193 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.510169    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-credential-socket\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.510316 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.510278    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-workload-socket\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.510437 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.510328    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-workload-certs\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.510530 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.510510    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istiod-ca-cert\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.512281 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.512262    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-envoy\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.512588 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.512568    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-podinfo\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.518780 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.518760    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbxzf\" (UniqueName: \"kubernetes.io/projected/d51b6769-c17d-4e9e-85f8-fd0e819273f8-kube-api-access-cbxzf\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.518780 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.518775    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-token\") pod \"e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") " pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.606179 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.606153    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:38.699648 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.699588    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:38.749929 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.749903    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"]
Jul 28 21:27:38.751878 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:27:38.751834    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51b6769_c17d_4e9e_85f8_fd0e819273f8.slice/crio-03bbc76e748476023597381e3a11574e43ca2b01fc85b11160de9346e2e41e7b WatchSource:0}: Error finding container 03bbc76e748476023597381e3a11574e43ca2b01fc85b11160de9346e2e41e7b: Status 404 returned error can't find the container with id 03bbc76e748476023597381e3a11574e43ca2b01fc85b11160de9346e2e41e7b
Jul 28 21:27:38.754148 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.754115    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:27:38.754268 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.754192    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:27:38.754268 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.754224    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:27:38.779501 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:38.779477    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw" event={"ID":"d51b6769-c17d-4e9e-85f8-fd0e819273f8","Type":"ContainerStarted","Data":"03bbc76e748476023597381e3a11574e43ca2b01fc85b11160de9346e2e41e7b"}
Jul 28 21:27:39.786276 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:39.786237    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw" event={"ID":"d51b6769-c17d-4e9e-85f8-fd0e819273f8","Type":"ContainerStarted","Data":"a1afa347d8143d966df99df767143add1e5c74d37811d61bf9f76f1613849ef7"}
Jul 28 21:27:39.808649 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:39.808600    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw" podStartSLOduration=1.8085862110000002 podStartE2EDuration="1.808586211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:27:39.806477646 +0000 UTC m=+2420.424059549" watchObservedRunningTime="2026-07-28 21:27:39.808586211 +0000 UTC m=+2420.426168104"
Jul 28 21:27:40.606499 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:40.606461    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:40.611417 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:40.611390    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:40.791242 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:40.791210    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:40.792235 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:40.792213    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:41.695278 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:41.695246    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:41.796973 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:41.796932    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:41.901050 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:41.900990    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:42.280655 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:42.280618    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:42.543505 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:42.543395    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:42.645934 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:42.645901    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:52.356679 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:52.356612    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:27:53.864136 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:53.864105    2537 generic.go:356] "Generic (PLEG): container finished" podID="d51b6769-c17d-4e9e-85f8-fd0e819273f8" containerID="a1afa347d8143d966df99df767143add1e5c74d37811d61bf9f76f1613849ef7" exitCode=0
Jul 28 21:27:53.864546 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:53.864177    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw" event={"ID":"d51b6769-c17d-4e9e-85f8-fd0e819273f8","Type":"ContainerDied","Data":"a1afa347d8143d966df99df767143add1e5c74d37811d61bf9f76f1613849ef7"}
Jul 28 21:27:53.864609 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:53.864592    2537 scope.go:117] "RemoveContainer" containerID="a1afa347d8143d966df99df767143add1e5c74d37811d61bf9f76f1613849ef7"
Jul 28 21:27:53.865585 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:53.865556    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:27:53.865680 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:53.865637    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:27:53.865743 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:53.865694    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:27:54.871319 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:54.871285    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw" event={"ID":"d51b6769-c17d-4e9e-85f8-fd0e819273f8","Type":"ContainerStarted","Data":"3e71a8f2262ad0e7371e3b55c35149e77f5ace797e38de49424931430bc344b6"}
Jul 28 21:27:55.606373 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:55.606336    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:55.607695 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:55.607660    2537 patch_prober.go:28] interesting pod/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 28 21:27:55.607846 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:55.607724    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw" podUID="d51b6769-c17d-4e9e-85f8-fd0e819273f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 28 21:27:55.791252 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:55.791217    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:56.607425 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:56.607388    2537 patch_prober.go:28] interesting pod/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 28 21:27:56.608055 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:56.608022    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw" podUID="d51b6769-c17d-4e9e-85f8-fd0e819273f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 28 21:27:57.607075 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:57.607036    2537 patch_prober.go:28] interesting pod/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 28 21:27:57.607488 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:57.607449    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw" podUID="d51b6769-c17d-4e9e-85f8-fd0e819273f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 28 21:27:58.610552 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:58.610513    2537 patch_prober.go:28] interesting pod/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 28 21:27:58.611047 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:58.610579    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw" podUID="d51b6769-c17d-4e9e-85f8-fd0e819273f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 28 21:27:59.611154 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:59.611125    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:27:59.611987 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:27:59.611969    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:28:06.666230 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:06.666182    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:28:07.001606 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:07.001481    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:28:07.290986 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:07.290878    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:28:07.874451 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:07.874419    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:28:09.785158 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:09.785119    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:28:09.871331 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:09.871284    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:28:09.933091 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:09.933052    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:28:10.187728 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.187686    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:28:10.472681 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.472584    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:28:10.567158 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.567110    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:28:10.595725 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.595690    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"]
Jul 28 21:28:10.596161 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.596146    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" containerName="istio-proxy"
Jul 28 21:28:10.596223 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.596163    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="0644cd8f-b07b-4c7a-b763-01b54daf7c03" containerName="istio-proxy"
Jul 28 21:28:10.599470 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.599454    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"
Jul 28 21:28:10.602056 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.602034    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-2c3f70\""
Jul 28 21:28:10.602192 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.602050    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-2c3f70-dockercfg-58mtj\""
Jul 28 21:28:10.608253 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.608227    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"]
Jul 28 21:28:10.692752 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.692716    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39eebe04-47c4-412a-a685-e0a8528a5ea2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6\" (UID: \"39eebe04-47c4-412a-a685-e0a8528a5ea2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"
Jul 28 21:28:10.692971 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.692826    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b5xr\" (UniqueName: \"kubernetes.io/projected/39eebe04-47c4-412a-a685-e0a8528a5ea2-kube-api-access-4b5xr\") pod \"payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6\" (UID: \"39eebe04-47c4-412a-a685-e0a8528a5ea2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"
Jul 28 21:28:10.793556 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.793466    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39eebe04-47c4-412a-a685-e0a8528a5ea2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6\" (UID: \"39eebe04-47c4-412a-a685-e0a8528a5ea2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"
Jul 28 21:28:10.793556 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.793536    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b5xr\" (UniqueName: \"kubernetes.io/projected/39eebe04-47c4-412a-a685-e0a8528a5ea2-kube-api-access-4b5xr\") pod \"payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6\" (UID: \"39eebe04-47c4-412a-a685-e0a8528a5ea2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"
Jul 28 21:28:10.794235 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.794214    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39eebe04-47c4-412a-a685-e0a8528a5ea2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6\" (UID: \"39eebe04-47c4-412a-a685-e0a8528a5ea2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"
Jul 28 21:28:10.802880 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.802835    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b5xr\" (UniqueName: \"kubernetes.io/projected/39eebe04-47c4-412a-a685-e0a8528a5ea2-kube-api-access-4b5xr\") pod \"payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6\" (UID: \"39eebe04-47c4-412a-a685-e0a8528a5ea2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"
Jul 28 21:28:10.911871 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:10.911828    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"
Jul 28 21:28:11.258867 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:11.258828    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"]
Jul 28 21:28:11.262152 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:28:11.262119    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39eebe04_47c4_412a_a685_e0a8528a5ea2.slice/crio-6c7b8123d832369d76def3c15f8eaf9e00685c4546372581ec71c44d3bd1757b WatchSource:0}: Error finding container 6c7b8123d832369d76def3c15f8eaf9e00685c4546372581ec71c44d3bd1757b: Status 404 returned error can't find the container with id 6c7b8123d832369d76def3c15f8eaf9e00685c4546372581ec71c44d3bd1757b
Jul 28 21:28:11.956630 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:11.956591    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6" event={"ID":"39eebe04-47c4-412a-a685-e0a8528a5ea2","Type":"ContainerStarted","Data":"a2c458ccc81b05efef294c2655e5399d67fe6a6d1a6bed9d760f81c59afbeb81"}
Jul 28 21:28:11.957047 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:11.956651    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6" event={"ID":"39eebe04-47c4-412a-a685-e0a8528a5ea2","Type":"ContainerStarted","Data":"6c7b8123d832369d76def3c15f8eaf9e00685c4546372581ec71c44d3bd1757b"}
Jul 28 21:28:11.957047 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:11.956697    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"
Jul 28 21:28:11.974017 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:11.973968    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6" podStartSLOduration=1.9739388930000001 podStartE2EDuration="1.973938893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:28:11.972380773 +0000 UTC m=+2452.589962676" watchObservedRunningTime="2026-07-28 21:28:11.973938893 +0000 UTC m=+2452.591520845"
Jul 28 21:28:22.963369 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:22.963331    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"
Jul 28 21:28:34.659746 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:34.659702    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:28:44.168830 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.168779    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"]
Jul 28 21:28:44.172514 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.172493    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.176380 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.176355    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1096ca\"/\"default-dockercfg-m8mbb\""
Jul 28 21:28:44.176513 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.176371    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1096ca\"/\"auth-test-model-1096ca-kserve-self-signed-certs\""
Jul 28 21:28:44.176513 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.176395    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1096ca\"/\"kube-root-ca.crt\""
Jul 28 21:28:44.176513 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.176355    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1096ca\"/\"openshift-service-ca.crt\""
Jul 28 21:28:44.181010 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.180815    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"]
Jul 28 21:28:44.198396 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.198367    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:28:44.289783 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.289753    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-dshm\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.289984 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.289794    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-tmp-dir\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.289984 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.289825    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-home\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.289984 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.289931    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2np58\" (UniqueName: \"kubernetes.io/projected/b652cfc6-abf3-4172-b4ab-8c60029963e1-kube-api-access-2np58\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.289984 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.289959    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-model-cache\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.290138 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.290028    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b652cfc6-abf3-4172-b4ab-8c60029963e1-tls-certs\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.390970 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.390927    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-dshm\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.390970 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.390977    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-tmp-dir\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.391214 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.391002    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-home\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.391214 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.391032    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2np58\" (UniqueName: \"kubernetes.io/projected/b652cfc6-abf3-4172-b4ab-8c60029963e1-kube-api-access-2np58\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.391214 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.391053    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-model-cache\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.391214 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.391094    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b652cfc6-abf3-4172-b4ab-8c60029963e1-tls-certs\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.391459 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.391437    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-tmp-dir\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.391532 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.391451    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-home\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.391532 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.391518    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-model-cache\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.393327 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.393306    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-dshm\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.393509 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.393492    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b652cfc6-abf3-4172-b4ab-8c60029963e1-tls-certs\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.399882 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.399844    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2np58\" (UniqueName: \"kubernetes.io/projected/b652cfc6-abf3-4172-b4ab-8c60029963e1-kube-api-access-2np58\") pod \"auth-test-model-1096ca-kserve-6f6bb96779-r9bh6\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") " pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.484140 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.484072    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:44.616888 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:44.616848    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"]
Jul 28 21:28:44.618878 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:28:44.618840    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb652cfc6_abf3_4172_b4ab_8c60029963e1.slice/crio-b7a41f7dcc26ea05e6c13862cbaa1c03444825fa70d54fd332b3a019fce9645c WatchSource:0}: Error finding container b7a41f7dcc26ea05e6c13862cbaa1c03444825fa70d54fd332b3a019fce9645c: Status 404 returned error can't find the container with id b7a41f7dcc26ea05e6c13862cbaa1c03444825fa70d54fd332b3a019fce9645c
Jul 28 21:28:45.115091 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:45.115049    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6" event={"ID":"b652cfc6-abf3-4172-b4ab-8c60029963e1","Type":"ContainerStarted","Data":"5a552cf91b6e28eb9dc8c55abe77495c75c466a9c54bc07f808bdb4d0240ab12"}
Jul 28 21:28:45.115091 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:45.115099    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6" event={"ID":"b652cfc6-abf3-4172-b4ab-8c60029963e1","Type":"ContainerStarted","Data":"b7a41f7dcc26ea05e6c13862cbaa1c03444825fa70d54fd332b3a019fce9645c"}
Jul 28 21:28:45.132374 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:45.132319    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6" podStartSLOduration=1.132299205 podStartE2EDuration="1.132299205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:28:45.131959533 +0000 UTC m=+2485.749541435" watchObservedRunningTime="2026-07-28 21:28:45.132299205 +0000 UTC m=+2485.749881154"
Jul 28 21:28:54.485026 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:54.484981    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:54.485458 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:54.485129    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:54.497922 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:54.497899    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:28:55.174435 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:28:55.174404    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:29:19.678566 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.678477    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"]
Jul 28 21:29:19.682974 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.682949    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.687495 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.687469    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"kube-root-ca.crt\""
Jul 28 21:29:19.688567 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.687483    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"auth-test-model-2c3f70-kserve-self-signed-certs\""
Jul 28 21:29:19.688567 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.687539    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"default-dockercfg-cvrsq\""
Jul 28 21:29:19.688567 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.688019    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"openshift-service-ca.crt\""
Jul 28 21:29:19.691137 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.691112    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"]
Jul 28 21:29:19.710962 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.710933    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:29:19.794848 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.794800    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-dshm\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.795036 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.794873    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-home\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.795036 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.794944    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-tmp-dir\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.795036 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.794976    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d79524af-a63c-4f46-82d9-804b3fde7b66-tls-certs\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.795167 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.795044    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2b5l\" (UniqueName: \"kubernetes.io/projected/d79524af-a63c-4f46-82d9-804b3fde7b66-kube-api-access-r2b5l\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.795167 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.795097    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-model-cache\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.895493 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.895456    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-tmp-dir\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.895493 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.895491    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d79524af-a63c-4f46-82d9-804b3fde7b66-tls-certs\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.895744 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.895522    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2b5l\" (UniqueName: \"kubernetes.io/projected/d79524af-a63c-4f46-82d9-804b3fde7b66-kube-api-access-r2b5l\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.895744 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.895539    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-model-cache\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.895744 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.895580    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-dshm\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.895744 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.895621    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-home\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.895991 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.895837    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-tmp-dir\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.895991 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.895928    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-model-cache\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.896092 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.896009    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-home\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.897821 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.897794    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-dshm\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.898546 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.898526    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"auth-test-model-2c3f70-kserve-self-signed-certs\""
Jul 28 21:29:19.904357 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.904332    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"kube-root-ca.crt\""
Jul 28 21:29:19.908850 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.908827    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d79524af-a63c-4f46-82d9-804b3fde7b66-tls-certs\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.915012 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.914993    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"openshift-service-ca.crt\""
Jul 28 21:29:19.925771 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.925746    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2b5l\" (UniqueName: \"kubernetes.io/projected/d79524af-a63c-4f46-82d9-804b3fde7b66-kube-api-access-r2b5l\") pod \"auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") " pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:19.999645 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:19.999573    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"default-dockercfg-cvrsq\""
Jul 28 21:29:20.007989 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:20.007958    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:20.147013 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:20.146943    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"]
Jul 28 21:29:20.149281 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:29:20.149255    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd79524af_a63c_4f46_82d9_804b3fde7b66.slice/crio-4f1d947a03f51ae99a17a9e3d692ddb50447bfbf9ddb0d7bc04cc76c273f4e63 WatchSource:0}: Error finding container 4f1d947a03f51ae99a17a9e3d692ddb50447bfbf9ddb0d7bc04cc76c273f4e63: Status 404 returned error can't find the container with id 4f1d947a03f51ae99a17a9e3d692ddb50447bfbf9ddb0d7bc04cc76c273f4e63
Jul 28 21:29:20.282967 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:20.282888    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj" event={"ID":"d79524af-a63c-4f46-82d9-804b3fde7b66","Type":"ContainerStarted","Data":"dbde53b2a946d8cd5bb86f0f2cde3b0ac1f845fb2db59205a38d6bfe4dfc67cc"}
Jul 28 21:29:20.282967 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:20.282941    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj" event={"ID":"d79524af-a63c-4f46-82d9-804b3fde7b66","Type":"ContainerStarted","Data":"4f1d947a03f51ae99a17a9e3d692ddb50447bfbf9ddb0d7bc04cc76c273f4e63"}
Jul 28 21:29:20.302318 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:20.302269    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj" podStartSLOduration=1.302253612 podStartE2EDuration="1.302253612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:29:20.300584873 +0000 UTC m=+2520.918166776" watchObservedRunningTime="2026-07-28 21:29:20.302253612 +0000 UTC m=+2520.919835513"
Jul 28 21:29:30.008929 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:30.008891    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:30.009420 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:30.008946    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:30.028496 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:30.028470    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:29:30.349093 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:29:30.349063    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:32:20.067777 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:32:20.067704    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:32:20.075638 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:32:20.075615    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:32:27.129088 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:32:27.129057    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:33:00.779798 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:00.779762    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:33:36.757615 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:36.757577    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"]
Jul 28 21:33:36.775398 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:36.775370    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:33:37.592287 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:37.592246    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"]
Jul 28 21:33:37.604328 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:37.604293    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:33:51.776575 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:51.776531    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6" podUID="b652cfc6-abf3-4172-b4ab-8c60029963e1" containerName="main" containerID="cri-o://5a552cf91b6e28eb9dc8c55abe77495c75c466a9c54bc07f808bdb4d0240ab12" gracePeriod=45
Jul 28 21:33:52.028575 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.028509    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:33:52.044556 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.044529    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-model-cache\") pod \"b652cfc6-abf3-4172-b4ab-8c60029963e1\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") "
Jul 28 21:33:52.044670 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.044561    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-home\") pod \"b652cfc6-abf3-4172-b4ab-8c60029963e1\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") "
Jul 28 21:33:52.044670 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.044612    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2np58\" (UniqueName: \"kubernetes.io/projected/b652cfc6-abf3-4172-b4ab-8c60029963e1-kube-api-access-2np58\") pod \"b652cfc6-abf3-4172-b4ab-8c60029963e1\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") "
Jul 28 21:33:52.044670 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.044631    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b652cfc6-abf3-4172-b4ab-8c60029963e1-tls-certs\") pod \"b652cfc6-abf3-4172-b4ab-8c60029963e1\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") "
Jul 28 21:33:52.044670 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.044662    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-tmp-dir\") pod \"b652cfc6-abf3-4172-b4ab-8c60029963e1\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") "
Jul 28 21:33:52.044907 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.044737    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-dshm\") pod \"b652cfc6-abf3-4172-b4ab-8c60029963e1\" (UID: \"b652cfc6-abf3-4172-b4ab-8c60029963e1\") "
Jul 28 21:33:52.044907 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.044782    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "b652cfc6-abf3-4172-b4ab-8c60029963e1" (UID: "b652cfc6-abf3-4172-b4ab-8c60029963e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:33:52.045019 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.044975    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-home" (OuterVolumeSpecName: "home") pod "b652cfc6-abf3-4172-b4ab-8c60029963e1" (UID: "b652cfc6-abf3-4172-b4ab-8c60029963e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:33:52.045019 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.044992    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b652cfc6-abf3-4172-b4ab-8c60029963e1" (UID: "b652cfc6-abf3-4172-b4ab-8c60029963e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:33:52.045149 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.045115    2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-tmp-dir\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:33:52.045149 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.045134    2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-model-cache\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:33:52.045149 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.045145    2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-home\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:33:52.046899 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.046872    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b652cfc6-abf3-4172-b4ab-8c60029963e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b652cfc6-abf3-4172-b4ab-8c60029963e1" (UID: "b652cfc6-abf3-4172-b4ab-8c60029963e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:33:52.047039 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.046934    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-dshm" (OuterVolumeSpecName: "dshm") pod "b652cfc6-abf3-4172-b4ab-8c60029963e1" (UID: "b652cfc6-abf3-4172-b4ab-8c60029963e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:33:52.047039 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.046951    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b652cfc6-abf3-4172-b4ab-8c60029963e1-kube-api-access-2np58" (OuterVolumeSpecName: "kube-api-access-2np58") pod "b652cfc6-abf3-4172-b4ab-8c60029963e1" (UID: "b652cfc6-abf3-4172-b4ab-8c60029963e1"). InnerVolumeSpecName "kube-api-access-2np58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:33:52.146184 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.146153    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2np58\" (UniqueName: \"kubernetes.io/projected/b652cfc6-abf3-4172-b4ab-8c60029963e1-kube-api-access-2np58\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:33:52.146184 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.146180    2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b652cfc6-abf3-4172-b4ab-8c60029963e1-tls-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:33:52.146184 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.146190    2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b652cfc6-abf3-4172-b4ab-8c60029963e1-dshm\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:33:52.611417 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.611355    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj" podUID="d79524af-a63c-4f46-82d9-804b3fde7b66" containerName="main" containerID="cri-o://dbde53b2a946d8cd5bb86f0f2cde3b0ac1f845fb2db59205a38d6bfe4dfc67cc" gracePeriod=45
Jul 28 21:33:52.644514 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.644485    2537 generic.go:356] "Generic (PLEG): container finished" podID="b652cfc6-abf3-4172-b4ab-8c60029963e1" containerID="5a552cf91b6e28eb9dc8c55abe77495c75c466a9c54bc07f808bdb4d0240ab12" exitCode=0
Jul 28 21:33:52.644624 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.644526    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"
Jul 28 21:33:52.644624 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.644564    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6" event={"ID":"b652cfc6-abf3-4172-b4ab-8c60029963e1","Type":"ContainerDied","Data":"5a552cf91b6e28eb9dc8c55abe77495c75c466a9c54bc07f808bdb4d0240ab12"}
Jul 28 21:33:52.644624 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.644608    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6" event={"ID":"b652cfc6-abf3-4172-b4ab-8c60029963e1","Type":"ContainerDied","Data":"b7a41f7dcc26ea05e6c13862cbaa1c03444825fa70d54fd332b3a019fce9645c"}
Jul 28 21:33:52.644732 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.644627    2537 scope.go:117] "RemoveContainer" containerID="5a552cf91b6e28eb9dc8c55abe77495c75c466a9c54bc07f808bdb4d0240ab12"
Jul 28 21:33:52.684845 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.684819    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"]
Jul 28 21:33:52.691081 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.691059    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/auth-test-model-1096ca-kserve-6f6bb96779-r9bh6"]
Jul 28 21:33:52.722524 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.722501    2537 scope.go:117] "RemoveContainer" containerID="5a552cf91b6e28eb9dc8c55abe77495c75c466a9c54bc07f808bdb4d0240ab12"
Jul 28 21:33:52.722851 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:33:52.722828    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a552cf91b6e28eb9dc8c55abe77495c75c466a9c54bc07f808bdb4d0240ab12\": container with ID starting with 5a552cf91b6e28eb9dc8c55abe77495c75c466a9c54bc07f808bdb4d0240ab12 not found: ID does not exist" containerID="5a552cf91b6e28eb9dc8c55abe77495c75c466a9c54bc07f808bdb4d0240ab12"
Jul 28 21:33:52.722942 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.722885    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a552cf91b6e28eb9dc8c55abe77495c75c466a9c54bc07f808bdb4d0240ab12"} err="failed to get container status \"5a552cf91b6e28eb9dc8c55abe77495c75c466a9c54bc07f808bdb4d0240ab12\": rpc error: code = NotFound desc = could not find container \"5a552cf91b6e28eb9dc8c55abe77495c75c466a9c54bc07f808bdb4d0240ab12\": container with ID starting with 5a552cf91b6e28eb9dc8c55abe77495c75c466a9c54bc07f808bdb4d0240ab12 not found: ID does not exist"
Jul 28 21:33:52.861497 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.861439    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:33:52.953738 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.953712    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-home\") pod \"d79524af-a63c-4f46-82d9-804b3fde7b66\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") "
Jul 28 21:33:52.953738 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.953747    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d79524af-a63c-4f46-82d9-804b3fde7b66-tls-certs\") pod \"d79524af-a63c-4f46-82d9-804b3fde7b66\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") "
Jul 28 21:33:52.953969 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.953766    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-model-cache\") pod \"d79524af-a63c-4f46-82d9-804b3fde7b66\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") "
Jul 28 21:33:52.953969 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.953788    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-dshm\") pod \"d79524af-a63c-4f46-82d9-804b3fde7b66\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") "
Jul 28 21:33:52.953969 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.953901    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-tmp-dir\") pod \"d79524af-a63c-4f46-82d9-804b3fde7b66\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") "
Jul 28 21:33:52.953969 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.953945    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2b5l\" (UniqueName: \"kubernetes.io/projected/d79524af-a63c-4f46-82d9-804b3fde7b66-kube-api-access-r2b5l\") pod \"d79524af-a63c-4f46-82d9-804b3fde7b66\" (UID: \"d79524af-a63c-4f46-82d9-804b3fde7b66\") "
Jul 28 21:33:52.954182 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.954045    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-model-cache" (OuterVolumeSpecName: "model-cache") pod "d79524af-a63c-4f46-82d9-804b3fde7b66" (UID: "d79524af-a63c-4f46-82d9-804b3fde7b66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:33:52.954182 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.954052    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-home" (OuterVolumeSpecName: "home") pod "d79524af-a63c-4f46-82d9-804b3fde7b66" (UID: "d79524af-a63c-4f46-82d9-804b3fde7b66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:33:52.954276 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.954193    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d79524af-a63c-4f46-82d9-804b3fde7b66" (UID: "d79524af-a63c-4f46-82d9-804b3fde7b66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:33:52.954276 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.954247    2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-home\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:33:52.954276 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.954267    2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-model-cache\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:33:52.956008 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.955985    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79524af-a63c-4f46-82d9-804b3fde7b66-kube-api-access-r2b5l" (OuterVolumeSpecName: "kube-api-access-r2b5l") pod "d79524af-a63c-4f46-82d9-804b3fde7b66" (UID: "d79524af-a63c-4f46-82d9-804b3fde7b66"). InnerVolumeSpecName "kube-api-access-r2b5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:33:52.956108 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.956013    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d79524af-a63c-4f46-82d9-804b3fde7b66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d79524af-a63c-4f46-82d9-804b3fde7b66" (UID: "d79524af-a63c-4f46-82d9-804b3fde7b66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:33:52.956108 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:52.956055    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-dshm" (OuterVolumeSpecName: "dshm") pod "d79524af-a63c-4f46-82d9-804b3fde7b66" (UID: "d79524af-a63c-4f46-82d9-804b3fde7b66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:33:53.055049 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:53.055017    2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-tmp-dir\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:33:53.055209 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:53.055134    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2b5l\" (UniqueName: \"kubernetes.io/projected/d79524af-a63c-4f46-82d9-804b3fde7b66-kube-api-access-r2b5l\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:33:53.055209 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:53.055162    2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d79524af-a63c-4f46-82d9-804b3fde7b66-tls-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:33:53.055209 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:53.055176    2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d79524af-a63c-4f46-82d9-804b3fde7b66-dshm\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:33:53.651479 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:53.651441    2537 generic.go:356] "Generic (PLEG): container finished" podID="d79524af-a63c-4f46-82d9-804b3fde7b66" containerID="dbde53b2a946d8cd5bb86f0f2cde3b0ac1f845fb2db59205a38d6bfe4dfc67cc" exitCode=0
Jul 28 21:33:53.651648 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:53.651486    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"
Jul 28 21:33:53.651648 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:53.651509    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj" event={"ID":"d79524af-a63c-4f46-82d9-804b3fde7b66","Type":"ContainerDied","Data":"dbde53b2a946d8cd5bb86f0f2cde3b0ac1f845fb2db59205a38d6bfe4dfc67cc"}
Jul 28 21:33:53.651648 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:53.651545    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj" event={"ID":"d79524af-a63c-4f46-82d9-804b3fde7b66","Type":"ContainerDied","Data":"4f1d947a03f51ae99a17a9e3d692ddb50447bfbf9ddb0d7bc04cc76c273f4e63"}
Jul 28 21:33:53.651648 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:53.651567    2537 scope.go:117] "RemoveContainer" containerID="dbde53b2a946d8cd5bb86f0f2cde3b0ac1f845fb2db59205a38d6bfe4dfc67cc"
Jul 28 21:33:53.661169 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:53.661151    2537 scope.go:117] "RemoveContainer" containerID="dbde53b2a946d8cd5bb86f0f2cde3b0ac1f845fb2db59205a38d6bfe4dfc67cc"
Jul 28 21:33:53.661428 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:33:53.661408    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbde53b2a946d8cd5bb86f0f2cde3b0ac1f845fb2db59205a38d6bfe4dfc67cc\": container with ID starting with dbde53b2a946d8cd5bb86f0f2cde3b0ac1f845fb2db59205a38d6bfe4dfc67cc not found: ID does not exist" containerID="dbde53b2a946d8cd5bb86f0f2cde3b0ac1f845fb2db59205a38d6bfe4dfc67cc"
Jul 28 21:33:53.661465 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:53.661436    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbde53b2a946d8cd5bb86f0f2cde3b0ac1f845fb2db59205a38d6bfe4dfc67cc"} err="failed to get container status \"dbde53b2a946d8cd5bb86f0f2cde3b0ac1f845fb2db59205a38d6bfe4dfc67cc\": rpc error: code = NotFound desc = could not find container \"dbde53b2a946d8cd5bb86f0f2cde3b0ac1f845fb2db59205a38d6bfe4dfc67cc\": container with ID starting with dbde53b2a946d8cd5bb86f0f2cde3b0ac1f845fb2db59205a38d6bfe4dfc67cc not found: ID does not exist"
Jul 28 21:33:53.674279 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:53.674254    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"]
Jul 28 21:33:53.680586 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:53.680565    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/auth-test-model-2c3f70-kserve-68c79cb69f-bfmzj"]
Jul 28 21:33:54.022384 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:54.022299    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b652cfc6-abf3-4172-b4ab-8c60029963e1" path="/var/lib/kubelet/pods/b652cfc6-abf3-4172-b4ab-8c60029963e1/volumes"
Jul 28 21:33:54.022747 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:54.022699    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d79524af-a63c-4f46-82d9-804b3fde7b66" path="/var/lib/kubelet/pods/d79524af-a63c-4f46-82d9-804b3fde7b66/volumes"
Jul 28 21:33:55.502323 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:55.502273    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:33:58.472537 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.472491    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"]
Jul 28 21:33:58.473102 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.473075    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b652cfc6-abf3-4172-b4ab-8c60029963e1" containerName="main"
Jul 28 21:33:58.473102 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.473103    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="b652cfc6-abf3-4172-b4ab-8c60029963e1" containerName="main"
Jul 28 21:33:58.473248 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.473123    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d79524af-a63c-4f46-82d9-804b3fde7b66" containerName="main"
Jul 28 21:33:58.473248 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.473131    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="d79524af-a63c-4f46-82d9-804b3fde7b66" containerName="main"
Jul 28 21:33:58.473348 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.473251    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="b652cfc6-abf3-4172-b4ab-8c60029963e1" containerName="main"
Jul 28 21:33:58.473348 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.473272    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="d79524af-a63c-4f46-82d9-804b3fde7b66" containerName="main"
Jul 28 21:33:58.477685 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.477664    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.481987 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.481941    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1096ca\"/\"openshift-service-ca.crt\""
Jul 28 21:33:58.481987 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.481980    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1096ca\"/\"sub-test-model-1096ca-kserve-self-signed-certs\""
Jul 28 21:33:58.482179 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.481986    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1096ca\"/\"default-dockercfg-m8mbb\""
Jul 28 21:33:58.482527 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.482507    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1096ca\"/\"kube-root-ca.crt\""
Jul 28 21:33:58.485658 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.485615    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"]
Jul 28 21:33:58.501185 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.501157    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-tmp-dir\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.501310 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.501192    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzhlq\" (UniqueName: \"kubernetes.io/projected/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-kube-api-access-zzhlq\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.501310 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.501228    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-home\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.501440 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.501349    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-dshm\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.501440 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.501408    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-tls-certs\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.501440 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.501432    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-model-cache\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.574652 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.574612    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:33:58.602344 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.602314    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-dshm\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.602476 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.602377    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-tls-certs\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.602476 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.602398    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-model-cache\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.602476 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.602434    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-tmp-dir\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.602648 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.602542    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzhlq\" (UniqueName: \"kubernetes.io/projected/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-kube-api-access-zzhlq\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.602648 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.602600    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-home\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.602786 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.602763    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-model-cache\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.602872 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.602830    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-tmp-dir\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.602976 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.602956    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-home\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.604849 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.604829    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-tls-certs\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.604952 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.604906    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-dshm\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.611848 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.611825    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzhlq\" (UniqueName: \"kubernetes.io/projected/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-kube-api-access-zzhlq\") pod \"sub-test-model-1096ca-kserve-b78dc99ff-x958s\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") " pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.789071 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.788993    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:33:58.933807 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:58.933784    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"]
Jul 28 21:33:58.935613 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:33:58.935581    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fbc8dd1_6165_42ee_a19f_7a70e68d6766.slice/crio-f85de05a8d18e0c8b3b9850a73268a391f7d376c09aebe9e6fde11bb04aec17c WatchSource:0}: Error finding container f85de05a8d18e0c8b3b9850a73268a391f7d376c09aebe9e6fde11bb04aec17c: Status 404 returned error can't find the container with id f85de05a8d18e0c8b3b9850a73268a391f7d376c09aebe9e6fde11bb04aec17c
Jul 28 21:33:59.681983 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:59.681944    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s" event={"ID":"4fbc8dd1-6165-42ee-a19f-7a70e68d6766","Type":"ContainerStarted","Data":"9668468a1375a90f2effd104fdc9fa6f2d87afd8cd6d0ad47d3a340c0023f3de"}
Jul 28 21:33:59.681983 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:59.681987    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s" event={"ID":"4fbc8dd1-6165-42ee-a19f-7a70e68d6766","Type":"ContainerStarted","Data":"f85de05a8d18e0c8b3b9850a73268a391f7d376c09aebe9e6fde11bb04aec17c"}
Jul 28 21:33:59.707636 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:33:59.707584    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s" podStartSLOduration=1.707570509 podStartE2EDuration="1.707570509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:33:59.706782435 +0000 UTC m=+2800.324364337" watchObservedRunningTime="2026-07-28 21:33:59.707570509 +0000 UTC m=+2800.325152484"
Jul 28 21:34:08.789682 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:34:08.789648    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:34:08.790279 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:34:08.789700    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:34:08.802445 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:34:08.802420    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:34:09.741341 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:34:09.741308    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:34:39.668734 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:34:39.668700    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:35:13.564673 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:13.564589    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:35:36.266633 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:36.266594    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"]
Jul 28 21:35:36.285026 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:36.284986    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:35:37.174559 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:37.174528    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:35:51.286974 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.286906    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s" podUID="4fbc8dd1-6165-42ee-a19f-7a70e68d6766" containerName="main" containerID="cri-o://9668468a1375a90f2effd104fdc9fa6f2d87afd8cd6d0ad47d3a340c0023f3de" gracePeriod=45
Jul 28 21:35:51.547549 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.547486    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:35:51.591016 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.590984    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-tmp-dir\") pod \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") "
Jul 28 21:35:51.591176 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.591027    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-home\") pod \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") "
Jul 28 21:35:51.591176 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.591064    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzhlq\" (UniqueName: \"kubernetes.io/projected/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-kube-api-access-zzhlq\") pod \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") "
Jul 28 21:35:51.591176 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.591142    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-dshm\") pod \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") "
Jul 28 21:35:51.591333 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.591196    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-tls-certs\") pod \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") "
Jul 28 21:35:51.591333 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.591244    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-model-cache\") pod \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\" (UID: \"4fbc8dd1-6165-42ee-a19f-7a70e68d6766\") "
Jul 28 21:35:51.591333 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.591279    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fbc8dd1-6165-42ee-a19f-7a70e68d6766" (UID: "4fbc8dd1-6165-42ee-a19f-7a70e68d6766"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:35:51.591495 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.591363    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-home" (OuterVolumeSpecName: "home") pod "4fbc8dd1-6165-42ee-a19f-7a70e68d6766" (UID: "4fbc8dd1-6165-42ee-a19f-7a70e68d6766"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:35:51.591584 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.591553    2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-tmp-dir\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:35:51.591584 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.591570    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fbc8dd1-6165-42ee-a19f-7a70e68d6766" (UID: "4fbc8dd1-6165-42ee-a19f-7a70e68d6766"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:35:51.591584 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.591577    2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-home\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:35:51.593466 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.593437    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fbc8dd1-6165-42ee-a19f-7a70e68d6766" (UID: "4fbc8dd1-6165-42ee-a19f-7a70e68d6766"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:35:51.593666 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.593628    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-dshm" (OuterVolumeSpecName: "dshm") pod "4fbc8dd1-6165-42ee-a19f-7a70e68d6766" (UID: "4fbc8dd1-6165-42ee-a19f-7a70e68d6766"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:35:51.593780 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.593762    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-kube-api-access-zzhlq" (OuterVolumeSpecName: "kube-api-access-zzhlq") pod "4fbc8dd1-6165-42ee-a19f-7a70e68d6766" (UID: "4fbc8dd1-6165-42ee-a19f-7a70e68d6766"). InnerVolumeSpecName "kube-api-access-zzhlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:35:51.692316 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.692283    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzhlq\" (UniqueName: \"kubernetes.io/projected/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-kube-api-access-zzhlq\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:35:51.692316 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.692312    2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-dshm\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:35:51.692316 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.692321    2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-tls-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:35:51.692552 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:51.692330    2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fbc8dd1-6165-42ee-a19f-7a70e68d6766-model-cache\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:35:52.254627 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:52.254594    2537 generic.go:356] "Generic (PLEG): container finished" podID="4fbc8dd1-6165-42ee-a19f-7a70e68d6766" containerID="9668468a1375a90f2effd104fdc9fa6f2d87afd8cd6d0ad47d3a340c0023f3de" exitCode=0
Jul 28 21:35:52.254803 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:52.254650    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"
Jul 28 21:35:52.254803 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:52.254687    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s" event={"ID":"4fbc8dd1-6165-42ee-a19f-7a70e68d6766","Type":"ContainerDied","Data":"9668468a1375a90f2effd104fdc9fa6f2d87afd8cd6d0ad47d3a340c0023f3de"}
Jul 28 21:35:52.254803 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:52.254727    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s" event={"ID":"4fbc8dd1-6165-42ee-a19f-7a70e68d6766","Type":"ContainerDied","Data":"f85de05a8d18e0c8b3b9850a73268a391f7d376c09aebe9e6fde11bb04aec17c"}
Jul 28 21:35:52.254803 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:52.254747    2537 scope.go:117] "RemoveContainer" containerID="9668468a1375a90f2effd104fdc9fa6f2d87afd8cd6d0ad47d3a340c0023f3de"
Jul 28 21:35:52.263756 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:52.263739    2537 scope.go:117] "RemoveContainer" containerID="9668468a1375a90f2effd104fdc9fa6f2d87afd8cd6d0ad47d3a340c0023f3de"
Jul 28 21:35:52.264060 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:35:52.264032    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9668468a1375a90f2effd104fdc9fa6f2d87afd8cd6d0ad47d3a340c0023f3de\": container with ID starting with 9668468a1375a90f2effd104fdc9fa6f2d87afd8cd6d0ad47d3a340c0023f3de not found: ID does not exist" containerID="9668468a1375a90f2effd104fdc9fa6f2d87afd8cd6d0ad47d3a340c0023f3de"
Jul 28 21:35:52.264129 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:52.264068    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9668468a1375a90f2effd104fdc9fa6f2d87afd8cd6d0ad47d3a340c0023f3de"} err="failed to get container status \"9668468a1375a90f2effd104fdc9fa6f2d87afd8cd6d0ad47d3a340c0023f3de\": rpc error: code = NotFound desc = could not find container \"9668468a1375a90f2effd104fdc9fa6f2d87afd8cd6d0ad47d3a340c0023f3de\": container with ID starting with 9668468a1375a90f2effd104fdc9fa6f2d87afd8cd6d0ad47d3a340c0023f3de not found: ID does not exist"
Jul 28 21:35:52.271441 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:52.271416    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"]
Jul 28 21:35:52.279758 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:52.279737    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/sub-test-model-1096ca-kserve-b78dc99ff-x958s"]
Jul 28 21:35:54.021546 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:35:54.021511    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fbc8dd1-6165-42ee-a19f-7a70e68d6766" path="/var/lib/kubelet/pods/4fbc8dd1-6165-42ee-a19f-7a70e68d6766/volumes"
Jul 28 21:36:08.867795 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.867764    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"]
Jul 28 21:36:08.868194 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.868183    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fbc8dd1-6165-42ee-a19f-7a70e68d6766" containerName="main"
Jul 28 21:36:08.868238 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.868197    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fbc8dd1-6165-42ee-a19f-7a70e68d6766" containerName="main"
Jul 28 21:36:08.868277 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.868272    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fbc8dd1-6165-42ee-a19f-7a70e68d6766" containerName="main"
Jul 28 21:36:08.871261 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.871242    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:08.875070 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.875046    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1096ca\"/\"default-dockercfg-m8mbb\""
Jul 28 21:36:08.875203 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.875069    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1096ca\"/\"openshift-service-ca.crt\""
Jul 28 21:36:08.875203 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.875102    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1096ca\"/\"kube-root-ca.crt\""
Jul 28 21:36:08.875203 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.875053    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1096ca\"/\"rate-test-model-1096ca-kserve-self-signed-certs\""
Jul 28 21:36:08.882667 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.882633    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"]
Jul 28 21:36:08.903276 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.903240    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:36:08.944688 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.944655    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864da9ba-7e9b-470a-8b95-0101e45494bd-tls-certs\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:08.944883 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.944731    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-tmp-dir\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:08.944883 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.944775    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw776\" (UniqueName: \"kubernetes.io/projected/864da9ba-7e9b-470a-8b95-0101e45494bd-kube-api-access-zw776\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:08.944883 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.944806    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-dshm\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:08.944883 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.944827    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-model-cache\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:08.944883 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:08.944846    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-home\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:09.045518 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:09.045474    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-tmp-dir\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:09.045721 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:09.045535    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw776\" (UniqueName: \"kubernetes.io/projected/864da9ba-7e9b-470a-8b95-0101e45494bd-kube-api-access-zw776\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:09.045721 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:09.045566    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-dshm\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:09.045721 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:09.045590    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-model-cache\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:09.045721 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:09.045607    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-home\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:09.045721 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:09.045652    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864da9ba-7e9b-470a-8b95-0101e45494bd-tls-certs\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:09.046027 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:09.045973    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-tmp-dir\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:09.046121 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:09.046100    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-model-cache\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:09.046203 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:09.046181    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-home\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:09.048458 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:09.048430    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864da9ba-7e9b-470a-8b95-0101e45494bd-tls-certs\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:09.048649 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:09.048628    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-dshm\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:09.054290 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:09.054268    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw776\" (UniqueName: \"kubernetes.io/projected/864da9ba-7e9b-470a-8b95-0101e45494bd-kube-api-access-zw776\") pod \"rate-test-model-1096ca-kserve-67f687cbcf-fp24d\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") " pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:09.182207 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:09.182116    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:09.316432 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:09.316400    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"]
Jul 28 21:36:09.319202 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:36:09.319177    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod864da9ba_7e9b_470a_8b95_0101e45494bd.slice/crio-a62d1e42d6bb07208dacefe07a93ff6af3d3823919adeff0225c46eb91fb2d3d WatchSource:0}: Error finding container a62d1e42d6bb07208dacefe07a93ff6af3d3823919adeff0225c46eb91fb2d3d: Status 404 returned error can't find the container with id a62d1e42d6bb07208dacefe07a93ff6af3d3823919adeff0225c46eb91fb2d3d
Jul 28 21:36:09.341585 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:09.341559    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d" event={"ID":"864da9ba-7e9b-470a-8b95-0101e45494bd","Type":"ContainerStarted","Data":"a62d1e42d6bb07208dacefe07a93ff6af3d3823919adeff0225c46eb91fb2d3d"}
Jul 28 21:36:10.352524 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:10.352482    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d" event={"ID":"864da9ba-7e9b-470a-8b95-0101e45494bd","Type":"ContainerStarted","Data":"09e95ad4fc59067d1f1e3fe6e50dae2608727057f67e2969dfdc7e00336624d5"}
Jul 28 21:36:10.371113 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:10.371047    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d" podStartSLOduration=2.371031158 podStartE2EDuration="2.371031158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:36:10.370132217 +0000 UTC m=+2930.987714121" watchObservedRunningTime="2026-07-28 21:36:10.371031158 +0000 UTC m=+2930.988613059"
Jul 28 21:36:19.182238 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:19.182204    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:19.182881 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:19.182254    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:19.194977 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:19.194954    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:19.408042 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:19.408010    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:36:45.186944 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:36:45.186834    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:37:20.097948 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:37:20.097921    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:37:20.106338 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:37:20.106317    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:37:53.721813 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:37:53.721773    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"]
Jul 28 21:37:53.745914 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:37:53.745883    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:37:54.991627 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:37:54.991597    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:08.741875 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:08.741815    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d" podUID="864da9ba-7e9b-470a-8b95-0101e45494bd" containerName="main" containerID="cri-o://09e95ad4fc59067d1f1e3fe6e50dae2608727057f67e2969dfdc7e00336624d5" gracePeriod=45
Jul 28 21:38:08.947412 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:08.947367    2537 generic.go:356] "Generic (PLEG): container finished" podID="864da9ba-7e9b-470a-8b95-0101e45494bd" containerID="09e95ad4fc59067d1f1e3fe6e50dae2608727057f67e2969dfdc7e00336624d5" exitCode=0
Jul 28 21:38:08.947563 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:08.947416    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d" event={"ID":"864da9ba-7e9b-470a-8b95-0101e45494bd","Type":"ContainerDied","Data":"09e95ad4fc59067d1f1e3fe6e50dae2608727057f67e2969dfdc7e00336624d5"}
Jul 28 21:38:09.002746 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.002692    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:38:09.098525 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.098493    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw776\" (UniqueName: \"kubernetes.io/projected/864da9ba-7e9b-470a-8b95-0101e45494bd-kube-api-access-zw776\") pod \"864da9ba-7e9b-470a-8b95-0101e45494bd\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") "
Jul 28 21:38:09.098686 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.098548    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864da9ba-7e9b-470a-8b95-0101e45494bd-tls-certs\") pod \"864da9ba-7e9b-470a-8b95-0101e45494bd\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") "
Jul 28 21:38:09.098686 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.098581    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-tmp-dir\") pod \"864da9ba-7e9b-470a-8b95-0101e45494bd\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") "
Jul 28 21:38:09.098686 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.098642    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-model-cache\") pod \"864da9ba-7e9b-470a-8b95-0101e45494bd\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") "
Jul 28 21:38:09.098686 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.098670    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-home\") pod \"864da9ba-7e9b-470a-8b95-0101e45494bd\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") "
Jul 28 21:38:09.098950 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.098707    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-dshm\") pod \"864da9ba-7e9b-470a-8b95-0101e45494bd\" (UID: \"864da9ba-7e9b-470a-8b95-0101e45494bd\") "
Jul 28 21:38:09.099004 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.098960    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "864da9ba-7e9b-470a-8b95-0101e45494bd" (UID: "864da9ba-7e9b-470a-8b95-0101e45494bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:38:09.099004 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.098974    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "864da9ba-7e9b-470a-8b95-0101e45494bd" (UID: "864da9ba-7e9b-470a-8b95-0101e45494bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:38:09.099079 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.098996    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-home" (OuterVolumeSpecName: "home") pod "864da9ba-7e9b-470a-8b95-0101e45494bd" (UID: "864da9ba-7e9b-470a-8b95-0101e45494bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:38:09.100694 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.100667    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864da9ba-7e9b-470a-8b95-0101e45494bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "864da9ba-7e9b-470a-8b95-0101e45494bd" (UID: "864da9ba-7e9b-470a-8b95-0101e45494bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:38:09.100800 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.100708    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864da9ba-7e9b-470a-8b95-0101e45494bd-kube-api-access-zw776" (OuterVolumeSpecName: "kube-api-access-zw776") pod "864da9ba-7e9b-470a-8b95-0101e45494bd" (UID: "864da9ba-7e9b-470a-8b95-0101e45494bd"). InnerVolumeSpecName "kube-api-access-zw776". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:38:09.100800 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.100714    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-dshm" (OuterVolumeSpecName: "dshm") pod "864da9ba-7e9b-470a-8b95-0101e45494bd" (UID: "864da9ba-7e9b-470a-8b95-0101e45494bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:38:09.200132 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.200103    2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-model-cache\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:38:09.200132 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.200129    2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-home\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:38:09.200295 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.200138    2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-dshm\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:38:09.200295 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.200148    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw776\" (UniqueName: \"kubernetes.io/projected/864da9ba-7e9b-470a-8b95-0101e45494bd-kube-api-access-zw776\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:38:09.200295 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.200159    2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864da9ba-7e9b-470a-8b95-0101e45494bd-tls-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:38:09.200295 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.200169    2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/864da9ba-7e9b-470a-8b95-0101e45494bd-tmp-dir\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:38:09.954187 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.954155    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"
Jul 28 21:38:09.954187 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.954170    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d" event={"ID":"864da9ba-7e9b-470a-8b95-0101e45494bd","Type":"ContainerDied","Data":"a62d1e42d6bb07208dacefe07a93ff6af3d3823919adeff0225c46eb91fb2d3d"}
Jul 28 21:38:09.954648 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.954214    2537 scope.go:117] "RemoveContainer" containerID="09e95ad4fc59067d1f1e3fe6e50dae2608727057f67e2969dfdc7e00336624d5"
Jul 28 21:38:09.975782 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.975758    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"]
Jul 28 21:38:09.979803 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:09.979780    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1096ca/rate-test-model-1096ca-kserve-67f687cbcf-fp24d"]
Jul 28 21:38:10.021466 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:10.021437    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="864da9ba-7e9b-470a-8b95-0101e45494bd" path="/var/lib/kubelet/pods/864da9ba-7e9b-470a-8b95-0101e45494bd/volumes"
Jul 28 21:38:14.088591 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:14.088502    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:14.391879 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:14.391790    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:14.492076 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:14.492045    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:17.620967 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:17.620931    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:17.712449 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:17.712411    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:17.778758 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:17.778728    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:18.101096 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.101057    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:18.411726 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.411640    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:18.516319 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.516282    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:18.524124 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.524097    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"]
Jul 28 21:38:18.524510 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.524495    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="864da9ba-7e9b-470a-8b95-0101e45494bd" containerName="main"
Jul 28 21:38:18.524575 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.524512    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="864da9ba-7e9b-470a-8b95-0101e45494bd" containerName="main"
Jul 28 21:38:18.524612 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.524586    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="864da9ba-7e9b-470a-8b95-0101e45494bd" containerName="main"
Jul 28 21:38:18.529081 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.529059    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"
Jul 28 21:38:18.531947 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.531923    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cf1484\""
Jul 28 21:38:18.532080 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.532059    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cf1484-dockercfg-8gdqp\""
Jul 28 21:38:18.538074 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.538047    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"]
Jul 28 21:38:18.585272 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.585245    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04abd142-5c8a-4567-966f-a41612901d82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz\" (UID: \"04abd142-5c8a-4567-966f-a41612901d82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"
Jul 28 21:38:18.585413 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.585283    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrk2t\" (UniqueName: \"kubernetes.io/projected/04abd142-5c8a-4567-966f-a41612901d82-kube-api-access-lrk2t\") pod \"payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz\" (UID: \"04abd142-5c8a-4567-966f-a41612901d82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"
Jul 28 21:38:18.686788 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.686702    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04abd142-5c8a-4567-966f-a41612901d82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz\" (UID: \"04abd142-5c8a-4567-966f-a41612901d82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"
Jul 28 21:38:18.686788 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.686748    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrk2t\" (UniqueName: \"kubernetes.io/projected/04abd142-5c8a-4567-966f-a41612901d82-kube-api-access-lrk2t\") pod \"payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz\" (UID: \"04abd142-5c8a-4567-966f-a41612901d82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"
Jul 28 21:38:18.687642 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.687618    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04abd142-5c8a-4567-966f-a41612901d82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz\" (UID: \"04abd142-5c8a-4567-966f-a41612901d82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"
Jul 28 21:38:18.695914 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.695890    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrk2t\" (UniqueName: \"kubernetes.io/projected/04abd142-5c8a-4567-966f-a41612901d82-kube-api-access-lrk2t\") pod \"payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz\" (UID: \"04abd142-5c8a-4567-966f-a41612901d82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"
Jul 28 21:38:18.840723 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:18.840687    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"
Jul 28 21:38:19.184501 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:19.184470    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"]
Jul 28 21:38:19.187020 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:38:19.186982    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04abd142_5c8a_4567_966f_a41612901d82.slice/crio-20db11b99b14d3d1b6c43e95dd645b9646408d940cf298db698f25098247f20f WatchSource:0}: Error finding container 20db11b99b14d3d1b6c43e95dd645b9646408d940cf298db698f25098247f20f: Status 404 returned error can't find the container with id 20db11b99b14d3d1b6c43e95dd645b9646408d940cf298db698f25098247f20f
Jul 28 21:38:20.004681 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:20.004632    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz" event={"ID":"04abd142-5c8a-4567-966f-a41612901d82","Type":"ContainerStarted","Data":"6c19371bc9dca9b611d18448e99a565d7996e2cdb191fdcdff2d385fa77a137b"}
Jul 28 21:38:20.005135 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:20.004735    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"
Jul 28 21:38:20.005135 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:20.004759    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz" event={"ID":"04abd142-5c8a-4567-966f-a41612901d82","Type":"ContainerStarted","Data":"20db11b99b14d3d1b6c43e95dd645b9646408d940cf298db698f25098247f20f"}
Jul 28 21:38:20.023580 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:20.023527    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz" podStartSLOduration=2.023514399 podStartE2EDuration="2.023514399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:38:20.02102229 +0000 UTC m=+3060.638604195" watchObservedRunningTime="2026-07-28 21:38:20.023514399 +0000 UTC m=+3060.641096333"
Jul 28 21:38:28.751704 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:28.751668    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:31.012845 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:31.012810    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"
Jul 28 21:38:42.596827 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:42.596790    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:43.277786 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.277750    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:43.546874 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.546779    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"]
Jul 28 21:38:43.550622 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.550586    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.553435 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.553413    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d69596-openshift-default-dockercfg-x6vgx\""
Jul 28 21:38:43.561750 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.561729    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"]
Jul 28 21:38:43.571093 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.571071    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:43.671342 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.671316    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:43.701196 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.701161    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-workload-certs\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.701384 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.701208    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-envoy\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.701384 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.701279    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-workload-socket\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.701522 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.701382    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istiod-ca-cert\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.701522 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.701425    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-data\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.701522 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.701467    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwx7f\" (UniqueName: \"kubernetes.io/projected/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-kube-api-access-rwx7f\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.701681 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.701533    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-token\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.701681 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.701617    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-credential-socket\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.701791 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.701707    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-podinfo\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.802605 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.802531    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istiod-ca-cert\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.802605 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.802569    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-data\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.802605 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.802600    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwx7f\" (UniqueName: \"kubernetes.io/projected/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-kube-api-access-rwx7f\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.802958 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.802636    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-token\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.802958 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.802692    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-credential-socket\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.802958 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.802725    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-podinfo\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.802958 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.802754    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-workload-certs\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.802958 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.802773    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-envoy\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.802958 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.802814    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-workload-socket\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.803300 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.803084    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-data\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.803300 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.803164    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-workload-certs\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.803300 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.803206    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-credential-socket\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.803300 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.803260    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-workload-socket\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.803444 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.803417    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istiod-ca-cert\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.805209 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.805182    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-envoy\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.805337 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.805319    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-podinfo\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.811075 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.811049    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwx7f\" (UniqueName: \"kubernetes.io/projected/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-kube-api-access-rwx7f\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.811167 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.811072    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-token\") pod \"e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") " pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:43.863099 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:43.863076    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:44.005755 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:44.005729    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"]
Jul 28 21:38:44.008392 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:38:44.008364    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606efe2f_25c8_4ad3_9fab_da5ad71f8b76.slice/crio-d8c88e6ef0a0c9038dcb92c6ed414a528db55bbdfa610f25e73f1b769dde75da WatchSource:0}: Error finding container d8c88e6ef0a0c9038dcb92c6ed414a528db55bbdfa610f25e73f1b769dde75da: Status 404 returned error can't find the container with id d8c88e6ef0a0c9038dcb92c6ed414a528db55bbdfa610f25e73f1b769dde75da
Jul 28 21:38:44.010686 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:44.010651    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:38:44.010779 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:44.010722    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:38:44.010779 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:44.010753    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:38:44.139115 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:44.139070    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz" event={"ID":"606efe2f-25c8-4ad3-9fab-da5ad71f8b76","Type":"ContainerStarted","Data":"644a3c65da0c628c976d003afe04c84f64cd93c33aae48ebb27a8d096e0d0bf8"}
Jul 28 21:38:44.139313 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:44.139145    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz" event={"ID":"606efe2f-25c8-4ad3-9fab-da5ad71f8b76","Type":"ContainerStarted","Data":"d8c88e6ef0a0c9038dcb92c6ed414a528db55bbdfa610f25e73f1b769dde75da"}
Jul 28 21:38:44.160270 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:44.160220    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz" podStartSLOduration=1.160203694 podStartE2EDuration="1.160203694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:38:44.158581749 +0000 UTC m=+3084.776163656" watchObservedRunningTime="2026-07-28 21:38:44.160203694 +0000 UTC m=+3084.777785595"
Jul 28 21:38:44.863631 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:44.863586    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:45.869252 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:45.869223    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:46.129312 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:46.129211    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:46.149676 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:46.149643    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:46.150518 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:46.150497    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:38:46.221124 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:46.221058    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:46.294621 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:46.294575    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:46.772052 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:46.772021    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:46.925420 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:46.925390    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:47.024758 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:47.024669    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:38:47.043746 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:47.043715    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"]
Jul 28 21:38:47.051087 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:47.051008    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"
Jul 28 21:38:47.054419 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:47.054399    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d69596\""
Jul 28 21:38:47.055176 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:47.055142    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d69596-dockercfg-52nwz\""
Jul 28 21:38:47.058572 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:47.058554    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"]
Jul 28 21:38:47.134806 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:47.134769    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c495140b-f5d2-4256-af03-bbd3db7b1e02-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn\" (UID: \"c495140b-f5d2-4256-af03-bbd3db7b1e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"
Jul 28 21:38:47.134995 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:47.134836    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpjrb\" (UniqueName: \"kubernetes.io/projected/c495140b-f5d2-4256-af03-bbd3db7b1e02-kube-api-access-mpjrb\") pod \"payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn\" (UID: \"c495140b-f5d2-4256-af03-bbd3db7b1e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"
Jul 28 21:38:47.236119 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:47.236079    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpjrb\" (UniqueName: \"kubernetes.io/projected/c495140b-f5d2-4256-af03-bbd3db7b1e02-kube-api-access-mpjrb\") pod \"payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn\" (UID: \"c495140b-f5d2-4256-af03-bbd3db7b1e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"
Jul 28 21:38:47.236275 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:47.236181    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c495140b-f5d2-4256-af03-bbd3db7b1e02-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn\" (UID: \"c495140b-f5d2-4256-af03-bbd3db7b1e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"
Jul 28 21:38:47.236821 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:47.236804    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c495140b-f5d2-4256-af03-bbd3db7b1e02-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn\" (UID: \"c495140b-f5d2-4256-af03-bbd3db7b1e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"
Jul 28 21:38:47.251553 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:47.251526    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpjrb\" (UniqueName: \"kubernetes.io/projected/c495140b-f5d2-4256-af03-bbd3db7b1e02-kube-api-access-mpjrb\") pod \"payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn\" (UID: \"c495140b-f5d2-4256-af03-bbd3db7b1e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"
Jul 28 21:38:47.364076 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:47.364043    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"
Jul 28 21:38:47.708304 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:47.708274    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"]
Jul 28 21:38:47.710289 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:38:47.710257    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc495140b_f5d2_4256_af03_bbd3db7b1e02.slice/crio-80ac1dee34ea01aa4d5a81348041890c1275737d6a5451444d2fb47dbd1c4036 WatchSource:0}: Error finding container 80ac1dee34ea01aa4d5a81348041890c1275737d6a5451444d2fb47dbd1c4036: Status 404 returned error can't find the container with id 80ac1dee34ea01aa4d5a81348041890c1275737d6a5451444d2fb47dbd1c4036
Jul 28 21:38:48.162253 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:48.162221    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn" event={"ID":"c495140b-f5d2-4256-af03-bbd3db7b1e02","Type":"ContainerStarted","Data":"1b20ee9f2de88cb05036c55452be6baa3d2870859aeb4198109574b8b8f55c25"}
Jul 28 21:38:48.162671 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:48.162273    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn" event={"ID":"c495140b-f5d2-4256-af03-bbd3db7b1e02","Type":"ContainerStarted","Data":"80ac1dee34ea01aa4d5a81348041890c1275737d6a5451444d2fb47dbd1c4036"}
Jul 28 21:38:48.181509 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:48.181464    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn" podStartSLOduration=1.181445807 podStartE2EDuration="1.181445807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:38:48.181075861 +0000 UTC m=+3088.798657765" watchObservedRunningTime="2026-07-28 21:38:48.181445807 +0000 UTC m=+3088.799027772"
Jul 28 21:38:49.167207 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:38:49.167175    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"
Jul 28 21:39:00.174768 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:00.174732    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"
Jul 28 21:39:02.191024 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:02.190986    2537 patch_prober.go:28] interesting pod/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 28 21:39:02.191429 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:02.191047    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz" podUID="606efe2f-25c8-4ad3-9fab-da5ad71f8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 21:39:03.249419 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:03.249343    2537 generic.go:356] "Generic (PLEG): container finished" podID="606efe2f-25c8-4ad3-9fab-da5ad71f8b76" containerID="644a3c65da0c628c976d003afe04c84f64cd93c33aae48ebb27a8d096e0d0bf8" exitCode=0
Jul 28 21:39:03.249419 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:03.249399    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz" event={"ID":"606efe2f-25c8-4ad3-9fab-da5ad71f8b76","Type":"ContainerDied","Data":"644a3c65da0c628c976d003afe04c84f64cd93c33aae48ebb27a8d096e0d0bf8"}
Jul 28 21:39:03.249849 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:03.249754    2537 scope.go:117] "RemoveContainer" containerID="644a3c65da0c628c976d003afe04c84f64cd93c33aae48ebb27a8d096e0d0bf8"
Jul 28 21:39:03.250907 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:03.250878    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:39:03.250981 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:03.250944    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:39:03.250981 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:03.250974    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 21:39:04.256721 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:04.256685    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz" event={"ID":"606efe2f-25c8-4ad3-9fab-da5ad71f8b76","Type":"ContainerStarted","Data":"a89063e4f0879d660aa3a5d852160fb90eaa63720dbf0460c08681a98937b966"}
Jul 28 21:39:04.863292 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:04.863254    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:39:04.863484 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:04.863359    2537 patch_prober.go:28] interesting pod/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Jul 28 21:39:04.863484 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:04.863415    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz" podUID="606efe2f-25c8-4ad3-9fab-da5ad71f8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Jul 28 21:39:05.864485 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:05.864445    2537 patch_prober.go:28] interesting pod/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Jul 28 21:39:05.864915 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:05.864515    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz" podUID="606efe2f-25c8-4ad3-9fab-da5ad71f8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Jul 28 21:39:06.864010 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:06.863970    2537 patch_prober.go:28] interesting pod/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Jul 28 21:39:06.864219 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:06.864030    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz" podUID="606efe2f-25c8-4ad3-9fab-da5ad71f8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Jul 28 21:39:07.863746 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:07.863710    2537 patch_prober.go:28] interesting pod/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Jul 28 21:39:07.864171 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:07.863775    2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz" podUID="606efe2f-25c8-4ad3-9fab-da5ad71f8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Jul 28 21:39:08.867892 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:08.867843    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:39:08.868257 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:08.868116    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:39:08.868691 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:08.868675    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:39:11.011472 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:11.011435    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:39:17.814675 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:17.814649    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-zjcmp_32df0cb1-de8b-4183-a6c5-9e0e3c03a67e/payload-processing/0.log"
Jul 28 21:39:30.572623 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.572586    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"]
Jul 28 21:39:30.577326 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.577309    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.580074 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.580046    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-cf1484\"/\"openshift-service-ca.crt\""
Jul 28 21:39:30.581333 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.581310    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-cf1484\"/\"ipp-route-cf1484-kserve-self-signed-certs\""
Jul 28 21:39:30.581455 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.581313    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-cf1484\"/\"kube-root-ca.crt\""
Jul 28 21:39:30.581455 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.581310    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-cf1484\"/\"default-dockercfg-68flm\""
Jul 28 21:39:30.585589 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.585359    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"]
Jul 28 21:39:30.587399 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.587375    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:39:30.745842 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.745804    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-tmp-dir\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.746022 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.745889    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-home\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.746022 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.745910    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-model-cache\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.746022 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.745928    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-tls-certs\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.746022 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.745954    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-dshm\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.746022 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.746013    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s5kz\" (UniqueName: \"kubernetes.io/projected/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-kube-api-access-9s5kz\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.846624 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.846532    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-home\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.846624 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.846575    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-model-cache\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.846624 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.846609    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-tls-certs\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.846960 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.846649    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-dshm\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.846960 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.846702    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s5kz\" (UniqueName: \"kubernetes.io/projected/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-kube-api-access-9s5kz\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.846960 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.846773    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-tmp-dir\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.847091 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.847014    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-home\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.847091 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.847031    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-model-cache\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.847390 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.847364    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-tmp-dir\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.849323 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.849301    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-dshm\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.849444 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.849413    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-tls-certs\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.856841 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.856816    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s5kz\" (UniqueName: \"kubernetes.io/projected/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-kube-api-access-9s5kz\") pod \"ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") " pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:30.888020 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:30.887986    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:31.026694 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:31.026670    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"]
Jul 28 21:39:31.028695 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:39:31.028670    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1432ae88_40f7_40ac_aeb7_f84c243e6f9b.slice/crio-bb8439d8451f7d71dab2d0ec561786d79cd9204ea35194e124400953856b1614 WatchSource:0}: Error finding container bb8439d8451f7d71dab2d0ec561786d79cd9204ea35194e124400953856b1614: Status 404 returned error can't find the container with id bb8439d8451f7d71dab2d0ec561786d79cd9204ea35194e124400953856b1614
Jul 28 21:39:31.397829 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:31.397787    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5" event={"ID":"1432ae88-40f7-40ac-aeb7-f84c243e6f9b","Type":"ContainerStarted","Data":"f0be24fdd8502dde12274400f774912fa9c3f1da236436e40d7046a6688fe37c"}
Jul 28 21:39:31.397829 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:31.397833    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5" event={"ID":"1432ae88-40f7-40ac-aeb7-f84c243e6f9b","Type":"ContainerStarted","Data":"bb8439d8451f7d71dab2d0ec561786d79cd9204ea35194e124400953856b1614"}
Jul 28 21:39:31.415295 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:31.415243    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5" podStartSLOduration=1.41522941 podStartE2EDuration="1.41522941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:39:31.414252654 +0000 UTC m=+3132.031834557" watchObservedRunningTime="2026-07-28 21:39:31.41522941 +0000 UTC m=+3132.032811312"
Jul 28 21:39:40.888718 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:40.888687    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:40.889175 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:40.888923    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:40.901555 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:40.901535    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:39:41.461446 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:39:41.461412    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:40:09.905738 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:09.905704    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-zjcmp_32df0cb1-de8b-4183-a6c5-9e0e3c03a67e/payload-processing/0.log"
Jul 28 21:40:10.284940 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:10.284828    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:10.587371 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:10.587320    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:11.286316 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:11.286282    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:13.181034 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:13.180997    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:13.262902 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:13.262838    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:13.325866 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:13.325806    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:13.425168 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:13.425133    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:13.763988 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:13.763937    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:13.865068 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:13.865033    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:13.979288 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:13.979252    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"]
Jul 28 21:40:13.983032 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:13.983013    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"
Jul 28 21:40:13.985930 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:13.985907    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-ea76b4\""
Jul 28 21:40:13.991774 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:13.991748    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"]
Jul 28 21:40:14.058636 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.058553    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"]
Jul 28 21:40:14.062228 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.062202    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"
Jul 28 21:40:14.065120 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.065100    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-ea76b4-dockercfg-h2z6w\""
Jul 28 21:40:14.070957 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.070934    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"]
Jul 28 21:40:14.144622 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.144582    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvcfm\" (UniqueName: \"kubernetes.io/projected/269e8a5c-e26c-4f24-bd40-1588baa67828-kube-api-access-lvcfm\") pod \"payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn\" (UID: \"269e8a5c-e26c-4f24-bd40-1588baa67828\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"
Jul 28 21:40:14.144622 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.144625    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5dqb\" (UniqueName: \"kubernetes.io/projected/71389d21-4bdc-4da0-8592-4ea8e3a7f4e3-kube-api-access-x5dqb\") pod \"payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt\" (UID: \"71389d21-4bdc-4da0-8592-4ea8e3a7f4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"
Jul 28 21:40:14.144926 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.144646    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71389d21-4bdc-4da0-8592-4ea8e3a7f4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt\" (UID: \"71389d21-4bdc-4da0-8592-4ea8e3a7f4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"
Jul 28 21:40:14.144926 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.144737    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269e8a5c-e26c-4f24-bd40-1588baa67828-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn\" (UID: \"269e8a5c-e26c-4f24-bd40-1588baa67828\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"
Jul 28 21:40:14.246090 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.246049    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvcfm\" (UniqueName: \"kubernetes.io/projected/269e8a5c-e26c-4f24-bd40-1588baa67828-kube-api-access-lvcfm\") pod \"payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn\" (UID: \"269e8a5c-e26c-4f24-bd40-1588baa67828\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"
Jul 28 21:40:14.246090 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.246095    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5dqb\" (UniqueName: \"kubernetes.io/projected/71389d21-4bdc-4da0-8592-4ea8e3a7f4e3-kube-api-access-x5dqb\") pod \"payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt\" (UID: \"71389d21-4bdc-4da0-8592-4ea8e3a7f4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"
Jul 28 21:40:14.246576 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.246112    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71389d21-4bdc-4da0-8592-4ea8e3a7f4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt\" (UID: \"71389d21-4bdc-4da0-8592-4ea8e3a7f4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"
Jul 28 21:40:14.246576 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.246296    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269e8a5c-e26c-4f24-bd40-1588baa67828-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn\" (UID: \"269e8a5c-e26c-4f24-bd40-1588baa67828\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"
Jul 28 21:40:14.246788 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.246769    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71389d21-4bdc-4da0-8592-4ea8e3a7f4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt\" (UID: \"71389d21-4bdc-4da0-8592-4ea8e3a7f4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"
Jul 28 21:40:14.246984 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.246966    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269e8a5c-e26c-4f24-bd40-1588baa67828-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn\" (UID: \"269e8a5c-e26c-4f24-bd40-1588baa67828\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"
Jul 28 21:40:14.259169 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.259142    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvcfm\" (UniqueName: \"kubernetes.io/projected/269e8a5c-e26c-4f24-bd40-1588baa67828-kube-api-access-lvcfm\") pod \"payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn\" (UID: \"269e8a5c-e26c-4f24-bd40-1588baa67828\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"
Jul 28 21:40:14.259277 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.259171    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5dqb\" (UniqueName: \"kubernetes.io/projected/71389d21-4bdc-4da0-8592-4ea8e3a7f4e3-kube-api-access-x5dqb\") pod \"payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt\" (UID: \"71389d21-4bdc-4da0-8592-4ea8e3a7f4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"
Jul 28 21:40:14.295316 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.295289    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"
Jul 28 21:40:14.374318 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.374282    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"
Jul 28 21:40:14.657275 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.657246    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"]
Jul 28 21:40:14.659206 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:40:14.659180    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71389d21_4bdc_4da0_8592_4ea8e3a7f4e3.slice/crio-736f04308bf9faa734f43aa068660deebcb170b3540f45513fcc6bd766c04bea WatchSource:0}: Error finding container 736f04308bf9faa734f43aa068660deebcb170b3540f45513fcc6bd766c04bea: Status 404 returned error can't find the container with id 736f04308bf9faa734f43aa068660deebcb170b3540f45513fcc6bd766c04bea
Jul 28 21:40:14.736837 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:14.736814    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"]
Jul 28 21:40:14.738593 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:40:14.738570    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269e8a5c_e26c_4f24_bd40_1588baa67828.slice/crio-f217a2d08d1419d4a3424002012e348f2d5016a627e4dd9f418fb4d4b66398f4 WatchSource:0}: Error finding container f217a2d08d1419d4a3424002012e348f2d5016a627e4dd9f418fb4d4b66398f4: Status 404 returned error can't find the container with id f217a2d08d1419d4a3424002012e348f2d5016a627e4dd9f418fb4d4b66398f4
Jul 28 21:40:15.628322 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:15.628281    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt" event={"ID":"71389d21-4bdc-4da0-8592-4ea8e3a7f4e3","Type":"ContainerStarted","Data":"338dd86da38f4f7c4446310fc4902772177a66c86e53bd49997e2178350c182f"}
Jul 28 21:40:15.628812 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:15.628350    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt" event={"ID":"71389d21-4bdc-4da0-8592-4ea8e3a7f4e3","Type":"ContainerStarted","Data":"736f04308bf9faa734f43aa068660deebcb170b3540f45513fcc6bd766c04bea"}
Jul 28 21:40:15.628812 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:15.628487    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"
Jul 28 21:40:15.630042 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:15.630019    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn" event={"ID":"269e8a5c-e26c-4f24-bd40-1588baa67828","Type":"ContainerStarted","Data":"e36c6f85f25cfeef4cef7559fd7b940bdf5c0592f59cb21ed1555fcf3814fdfa"}
Jul 28 21:40:15.630188 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:15.630056    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn" event={"ID":"269e8a5c-e26c-4f24-bd40-1588baa67828","Type":"ContainerStarted","Data":"f217a2d08d1419d4a3424002012e348f2d5016a627e4dd9f418fb4d4b66398f4"}
Jul 28 21:40:15.630188 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:15.630142    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"
Jul 28 21:40:15.646655 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:15.646617    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt" podStartSLOduration=2.646605242 podStartE2EDuration="2.646605242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:40:15.644132295 +0000 UTC m=+3176.261714196" watchObservedRunningTime="2026-07-28 21:40:15.646605242 +0000 UTC m=+3176.264187168"
Jul 28 21:40:15.661372 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:15.661322    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn" podStartSLOduration=1.661308781 podStartE2EDuration="1.661308781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:40:15.659413858 +0000 UTC m=+3176.276995762" watchObservedRunningTime="2026-07-28 21:40:15.661308781 +0000 UTC m=+3176.278890683"
Jul 28 21:40:26.637753 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:26.637724    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"
Jul 28 21:40:26.638167 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:26.637910    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"
Jul 28 21:40:38.159739 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:38.159692    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:43.896520 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:43.896478    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:43.930264 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:43.930229    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"]
Jul 28 21:40:43.930527 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:43.930483    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn" podUID="269e8a5c-e26c-4f24-bd40-1588baa67828" containerName="payload-processing" containerID="cri-o://e36c6f85f25cfeef4cef7559fd7b940bdf5c0592f59cb21ed1555fcf3814fdfa" gracePeriod=30
Jul 28 21:40:43.935579 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:43.935553    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"]
Jul 28 21:40:43.935800 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:43.935776    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt" podUID="71389d21-4bdc-4da0-8592-4ea8e3a7f4e3" containerName="payload-pre-processing" containerID="cri-o://338dd86da38f4f7c4446310fc4902772177a66c86e53bd49997e2178350c182f" gracePeriod=30
Jul 28 21:40:43.996249 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:43.996217    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:44.065558 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.065523    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:44.120617 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.120578    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:44.228959 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.228935    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"
Jul 28 21:40:44.231999 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.231980    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"
Jul 28 21:40:44.326057 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.326025    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71389d21-4bdc-4da0-8592-4ea8e3a7f4e3-plugins-config-volume\") pod \"71389d21-4bdc-4da0-8592-4ea8e3a7f4e3\" (UID: \"71389d21-4bdc-4da0-8592-4ea8e3a7f4e3\") "
Jul 28 21:40:44.326057 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.326061    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5dqb\" (UniqueName: \"kubernetes.io/projected/71389d21-4bdc-4da0-8592-4ea8e3a7f4e3-kube-api-access-x5dqb\") pod \"71389d21-4bdc-4da0-8592-4ea8e3a7f4e3\" (UID: \"71389d21-4bdc-4da0-8592-4ea8e3a7f4e3\") "
Jul 28 21:40:44.326306 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.326108    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvcfm\" (UniqueName: \"kubernetes.io/projected/269e8a5c-e26c-4f24-bd40-1588baa67828-kube-api-access-lvcfm\") pod \"269e8a5c-e26c-4f24-bd40-1588baa67828\" (UID: \"269e8a5c-e26c-4f24-bd40-1588baa67828\") "
Jul 28 21:40:44.326306 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.326124    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269e8a5c-e26c-4f24-bd40-1588baa67828-plugins-config-volume\") pod \"269e8a5c-e26c-4f24-bd40-1588baa67828\" (UID: \"269e8a5c-e26c-4f24-bd40-1588baa67828\") "
Jul 28 21:40:44.326488 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.326462    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71389d21-4bdc-4da0-8592-4ea8e3a7f4e3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71389d21-4bdc-4da0-8592-4ea8e3a7f4e3" (UID: "71389d21-4bdc-4da0-8592-4ea8e3a7f4e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:40:44.326546 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.326510    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269e8a5c-e26c-4f24-bd40-1588baa67828-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "269e8a5c-e26c-4f24-bd40-1588baa67828" (UID: "269e8a5c-e26c-4f24-bd40-1588baa67828"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:40:44.328279 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.328255    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269e8a5c-e26c-4f24-bd40-1588baa67828-kube-api-access-lvcfm" (OuterVolumeSpecName: "kube-api-access-lvcfm") pod "269e8a5c-e26c-4f24-bd40-1588baa67828" (UID: "269e8a5c-e26c-4f24-bd40-1588baa67828"). InnerVolumeSpecName "kube-api-access-lvcfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:40:44.328323 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.328277    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71389d21-4bdc-4da0-8592-4ea8e3a7f4e3-kube-api-access-x5dqb" (OuterVolumeSpecName: "kube-api-access-x5dqb") pod "71389d21-4bdc-4da0-8592-4ea8e3a7f4e3" (UID: "71389d21-4bdc-4da0-8592-4ea8e3a7f4e3"). InnerVolumeSpecName "kube-api-access-x5dqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:40:44.427288 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.427208    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvcfm\" (UniqueName: \"kubernetes.io/projected/269e8a5c-e26c-4f24-bd40-1588baa67828-kube-api-access-lvcfm\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:40:44.427288 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.427236    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269e8a5c-e26c-4f24-bd40-1588baa67828-plugins-config-volume\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:40:44.427288 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.427246    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71389d21-4bdc-4da0-8592-4ea8e3a7f4e3-plugins-config-volume\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:40:44.427288 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.427254    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5dqb\" (UniqueName: \"kubernetes.io/projected/71389d21-4bdc-4da0-8592-4ea8e3a7f4e3-kube-api-access-x5dqb\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:40:44.429207 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.429181    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:44.796106 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.796007    2537 generic.go:356] "Generic (PLEG): container finished" podID="71389d21-4bdc-4da0-8592-4ea8e3a7f4e3" containerID="338dd86da38f4f7c4446310fc4902772177a66c86e53bd49997e2178350c182f" exitCode=0
Jul 28 21:40:44.796106 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.796052    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"
Jul 28 21:40:44.796338 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.796101    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt" event={"ID":"71389d21-4bdc-4da0-8592-4ea8e3a7f4e3","Type":"ContainerDied","Data":"338dd86da38f4f7c4446310fc4902772177a66c86e53bd49997e2178350c182f"}
Jul 28 21:40:44.796338 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.796158    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt" event={"ID":"71389d21-4bdc-4da0-8592-4ea8e3a7f4e3","Type":"ContainerDied","Data":"736f04308bf9faa734f43aa068660deebcb170b3540f45513fcc6bd766c04bea"}
Jul 28 21:40:44.796338 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.796183    2537 scope.go:117] "RemoveContainer" containerID="338dd86da38f4f7c4446310fc4902772177a66c86e53bd49997e2178350c182f"
Jul 28 21:40:44.797470 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.797444    2537 generic.go:356] "Generic (PLEG): container finished" podID="269e8a5c-e26c-4f24-bd40-1588baa67828" containerID="e36c6f85f25cfeef4cef7559fd7b940bdf5c0592f59cb21ed1555fcf3814fdfa" exitCode=0
Jul 28 21:40:44.797600 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.797488    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"
Jul 28 21:40:44.797600 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.797535    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn" event={"ID":"269e8a5c-e26c-4f24-bd40-1588baa67828","Type":"ContainerDied","Data":"e36c6f85f25cfeef4cef7559fd7b940bdf5c0592f59cb21ed1555fcf3814fdfa"}
Jul 28 21:40:44.797600 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.797590    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn" event={"ID":"269e8a5c-e26c-4f24-bd40-1588baa67828","Type":"ContainerDied","Data":"f217a2d08d1419d4a3424002012e348f2d5016a627e4dd9f418fb4d4b66398f4"}
Jul 28 21:40:44.809204 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.809183    2537 scope.go:117] "RemoveContainer" containerID="338dd86da38f4f7c4446310fc4902772177a66c86e53bd49997e2178350c182f"
Jul 28 21:40:44.809667 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:40:44.809632    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338dd86da38f4f7c4446310fc4902772177a66c86e53bd49997e2178350c182f\": container with ID starting with 338dd86da38f4f7c4446310fc4902772177a66c86e53bd49997e2178350c182f not found: ID does not exist" containerID="338dd86da38f4f7c4446310fc4902772177a66c86e53bd49997e2178350c182f"
Jul 28 21:40:44.809934 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.809672    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338dd86da38f4f7c4446310fc4902772177a66c86e53bd49997e2178350c182f"} err="failed to get container status \"338dd86da38f4f7c4446310fc4902772177a66c86e53bd49997e2178350c182f\": rpc error: code = NotFound desc = could not find container \"338dd86da38f4f7c4446310fc4902772177a66c86e53bd49997e2178350c182f\": container with ID starting with 338dd86da38f4f7c4446310fc4902772177a66c86e53bd49997e2178350c182f not found: ID does not exist"
Jul 28 21:40:44.809934 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.809696    2537 scope.go:117] "RemoveContainer" containerID="e36c6f85f25cfeef4cef7559fd7b940bdf5c0592f59cb21ed1555fcf3814fdfa"
Jul 28 21:40:44.821543 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.821525    2537 scope.go:117] "RemoveContainer" containerID="e36c6f85f25cfeef4cef7559fd7b940bdf5c0592f59cb21ed1555fcf3814fdfa"
Jul 28 21:40:44.821807 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:40:44.821786    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36c6f85f25cfeef4cef7559fd7b940bdf5c0592f59cb21ed1555fcf3814fdfa\": container with ID starting with e36c6f85f25cfeef4cef7559fd7b940bdf5c0592f59cb21ed1555fcf3814fdfa not found: ID does not exist" containerID="e36c6f85f25cfeef4cef7559fd7b940bdf5c0592f59cb21ed1555fcf3814fdfa"
Jul 28 21:40:44.821888 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.821815    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36c6f85f25cfeef4cef7559fd7b940bdf5c0592f59cb21ed1555fcf3814fdfa"} err="failed to get container status \"e36c6f85f25cfeef4cef7559fd7b940bdf5c0592f59cb21ed1555fcf3814fdfa\": rpc error: code = NotFound desc = could not find container \"e36c6f85f25cfeef4cef7559fd7b940bdf5c0592f59cb21ed1555fcf3814fdfa\": container with ID starting with e36c6f85f25cfeef4cef7559fd7b940bdf5c0592f59cb21ed1555fcf3814fdfa not found: ID does not exist"
Jul 28 21:40:44.826427 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.826404    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"]
Jul 28 21:40:44.830028 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.830005    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ea76b4-6df5444c98-kkvpn"]
Jul 28 21:40:44.843098 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.843078    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"]
Jul 28 21:40:44.849726 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:44.849702    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ea76b4-5955676448-7zhxt"]
Jul 28 21:40:46.021768 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:46.021732    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="269e8a5c-e26c-4f24-bd40-1588baa67828" path="/var/lib/kubelet/pods/269e8a5c-e26c-4f24-bd40-1588baa67828/volumes"
Jul 28 21:40:46.022173 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:46.022131    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71389d21-4bdc-4da0-8592-4ea8e3a7f4e3" path="/var/lib/kubelet/pods/71389d21-4bdc-4da0-8592-4ea8e3a7f4e3/volumes"
Jul 28 21:40:50.240276 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:50.240242    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:50.340961 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:50.340912    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:50.513805 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:50.513721    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:50.728449 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:50.728413    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:51.981799 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:51.981749    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:40:52.052850 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:40:52.052816    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:09.809329 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:09.809291    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:09.844337 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:09.844306    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"]
Jul 28 21:41:09.844580 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:09.844555    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz" podUID="04abd142-5c8a-4567-966f-a41612901d82" containerName="payload-pre-processing" containerID="cri-o://6c19371bc9dca9b611d18448e99a565d7996e2cdb191fdcdff2d385fa77a137b" gracePeriod=30
Jul 28 21:41:09.870312 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:09.870283    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:09.925710 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:09.925678    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:09.980608 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:09.980574    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:10.114841 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.114817    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"
Jul 28 21:41:10.267560 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.267529    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrk2t\" (UniqueName: \"kubernetes.io/projected/04abd142-5c8a-4567-966f-a41612901d82-kube-api-access-lrk2t\") pod \"04abd142-5c8a-4567-966f-a41612901d82\" (UID: \"04abd142-5c8a-4567-966f-a41612901d82\") "
Jul 28 21:41:10.267718 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.267568    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04abd142-5c8a-4567-966f-a41612901d82-plugins-config-volume\") pod \"04abd142-5c8a-4567-966f-a41612901d82\" (UID: \"04abd142-5c8a-4567-966f-a41612901d82\") "
Jul 28 21:41:10.267998 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.267974    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04abd142-5c8a-4567-966f-a41612901d82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "04abd142-5c8a-4567-966f-a41612901d82" (UID: "04abd142-5c8a-4567-966f-a41612901d82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:41:10.269626 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.269602    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04abd142-5c8a-4567-966f-a41612901d82-kube-api-access-lrk2t" (OuterVolumeSpecName: "kube-api-access-lrk2t") pod "04abd142-5c8a-4567-966f-a41612901d82" (UID: "04abd142-5c8a-4567-966f-a41612901d82"). InnerVolumeSpecName "kube-api-access-lrk2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:41:10.355543 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.355470    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:10.368785 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.368756    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrk2t\" (UniqueName: \"kubernetes.io/projected/04abd142-5c8a-4567-966f-a41612901d82-kube-api-access-lrk2t\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:41:10.368785 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.368785    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04abd142-5c8a-4567-966f-a41612901d82-plugins-config-volume\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:41:10.933802 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.933760    2537 generic.go:356] "Generic (PLEG): container finished" podID="04abd142-5c8a-4567-966f-a41612901d82" containerID="6c19371bc9dca9b611d18448e99a565d7996e2cdb191fdcdff2d385fa77a137b" exitCode=0
Jul 28 21:41:10.933802 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.933806    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"
Jul 28 21:41:10.934276 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.933799    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz" event={"ID":"04abd142-5c8a-4567-966f-a41612901d82","Type":"ContainerDied","Data":"6c19371bc9dca9b611d18448e99a565d7996e2cdb191fdcdff2d385fa77a137b"}
Jul 28 21:41:10.934276 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.933939    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz" event={"ID":"04abd142-5c8a-4567-966f-a41612901d82","Type":"ContainerDied","Data":"20db11b99b14d3d1b6c43e95dd645b9646408d940cf298db698f25098247f20f"}
Jul 28 21:41:10.934276 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.933969    2537 scope.go:117] "RemoveContainer" containerID="6c19371bc9dca9b611d18448e99a565d7996e2cdb191fdcdff2d385fa77a137b"
Jul 28 21:41:10.943448 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.943433    2537 scope.go:117] "RemoveContainer" containerID="6c19371bc9dca9b611d18448e99a565d7996e2cdb191fdcdff2d385fa77a137b"
Jul 28 21:41:10.943711 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:41:10.943692    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c19371bc9dca9b611d18448e99a565d7996e2cdb191fdcdff2d385fa77a137b\": container with ID starting with 6c19371bc9dca9b611d18448e99a565d7996e2cdb191fdcdff2d385fa77a137b not found: ID does not exist" containerID="6c19371bc9dca9b611d18448e99a565d7996e2cdb191fdcdff2d385fa77a137b"
Jul 28 21:41:10.943748 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.943719    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c19371bc9dca9b611d18448e99a565d7996e2cdb191fdcdff2d385fa77a137b"} err="failed to get container status \"6c19371bc9dca9b611d18448e99a565d7996e2cdb191fdcdff2d385fa77a137b\": rpc error: code = NotFound desc = could not find container \"6c19371bc9dca9b611d18448e99a565d7996e2cdb191fdcdff2d385fa77a137b\": container with ID starting with 6c19371bc9dca9b611d18448e99a565d7996e2cdb191fdcdff2d385fa77a137b not found: ID does not exist"
Jul 28 21:41:10.955395 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.955373    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"]
Jul 28 21:41:10.961228 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:10.961204    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf1484-6c4697694d-7x6rz"]
Jul 28 21:41:12.020814 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:12.020780    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04abd142-5c8a-4567-966f-a41612901d82" path="/var/lib/kubelet/pods/04abd142-5c8a-4567-966f-a41612901d82/volumes"
Jul 28 21:41:14.780585 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:14.780505    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:15.341364 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:15.341330    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"]
Jul 28 21:41:15.413000 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:15.412961    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:30.360574 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.360529    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5" podUID="1432ae88-40f7-40ac-aeb7-f84c243e6f9b" containerName="main" containerID="cri-o://f0be24fdd8502dde12274400f774912fa9c3f1da236436e40d7046a6688fe37c" gracePeriod=45
Jul 28 21:41:30.621422 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.621366    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:41:30.758994 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.758963    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-tls-certs\") pod \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") "
Jul 28 21:41:30.759153 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.759025    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-dshm\") pod \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") "
Jul 28 21:41:30.759153 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.759075    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-home\") pod \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") "
Jul 28 21:41:30.759153 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.759146    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s5kz\" (UniqueName: \"kubernetes.io/projected/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-kube-api-access-9s5kz\") pod \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") "
Jul 28 21:41:30.759294 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.759188    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-tmp-dir\") pod \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") "
Jul 28 21:41:30.759294 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.759220    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-model-cache\") pod \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\" (UID: \"1432ae88-40f7-40ac-aeb7-f84c243e6f9b\") "
Jul 28 21:41:30.759464 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.759442    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-home" (OuterVolumeSpecName: "home") pod "1432ae88-40f7-40ac-aeb7-f84c243e6f9b" (UID: "1432ae88-40f7-40ac-aeb7-f84c243e6f9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:41:30.759634 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.759612    2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-home\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:41:30.759695 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.759635    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "1432ae88-40f7-40ac-aeb7-f84c243e6f9b" (UID: "1432ae88-40f7-40ac-aeb7-f84c243e6f9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:41:30.759695 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.759630    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1432ae88-40f7-40ac-aeb7-f84c243e6f9b" (UID: "1432ae88-40f7-40ac-aeb7-f84c243e6f9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:41:30.761253 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.761224    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-dshm" (OuterVolumeSpecName: "dshm") pod "1432ae88-40f7-40ac-aeb7-f84c243e6f9b" (UID: "1432ae88-40f7-40ac-aeb7-f84c243e6f9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:41:30.761356 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.761328    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1432ae88-40f7-40ac-aeb7-f84c243e6f9b" (UID: "1432ae88-40f7-40ac-aeb7-f84c243e6f9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:41:30.761442 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.761425    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-kube-api-access-9s5kz" (OuterVolumeSpecName: "kube-api-access-9s5kz") pod "1432ae88-40f7-40ac-aeb7-f84c243e6f9b" (UID: "1432ae88-40f7-40ac-aeb7-f84c243e6f9b"). InnerVolumeSpecName "kube-api-access-9s5kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:41:30.861073 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.861047    2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-model-cache\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:41:30.861073 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.861073    2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-tls-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:41:30.861250 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.861087    2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-dshm\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:41:30.861250 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.861106    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s5kz\" (UniqueName: \"kubernetes.io/projected/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-kube-api-access-9s5kz\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:41:30.861250 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:30.861117    2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1432ae88-40f7-40ac-aeb7-f84c243e6f9b-tmp-dir\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:41:31.036051 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:31.035954    2537 generic.go:356] "Generic (PLEG): container finished" podID="1432ae88-40f7-40ac-aeb7-f84c243e6f9b" containerID="f0be24fdd8502dde12274400f774912fa9c3f1da236436e40d7046a6688fe37c" exitCode=0
Jul 28 21:41:31.036051 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:31.035992    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"
Jul 28 21:41:31.036051 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:31.036024    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5" event={"ID":"1432ae88-40f7-40ac-aeb7-f84c243e6f9b","Type":"ContainerDied","Data":"f0be24fdd8502dde12274400f774912fa9c3f1da236436e40d7046a6688fe37c"}
Jul 28 21:41:31.036299 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:31.036061    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5" event={"ID":"1432ae88-40f7-40ac-aeb7-f84c243e6f9b","Type":"ContainerDied","Data":"bb8439d8451f7d71dab2d0ec561786d79cd9204ea35194e124400953856b1614"}
Jul 28 21:41:31.036299 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:31.036084    2537 scope.go:117] "RemoveContainer" containerID="f0be24fdd8502dde12274400f774912fa9c3f1da236436e40d7046a6688fe37c"
Jul 28 21:41:31.045237 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:31.045221    2537 scope.go:117] "RemoveContainer" containerID="f0be24fdd8502dde12274400f774912fa9c3f1da236436e40d7046a6688fe37c"
Jul 28 21:41:31.045469 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:41:31.045450    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0be24fdd8502dde12274400f774912fa9c3f1da236436e40d7046a6688fe37c\": container with ID starting with f0be24fdd8502dde12274400f774912fa9c3f1da236436e40d7046a6688fe37c not found: ID does not exist" containerID="f0be24fdd8502dde12274400f774912fa9c3f1da236436e40d7046a6688fe37c"
Jul 28 21:41:31.045537 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:31.045477    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0be24fdd8502dde12274400f774912fa9c3f1da236436e40d7046a6688fe37c"} err="failed to get container status \"f0be24fdd8502dde12274400f774912fa9c3f1da236436e40d7046a6688fe37c\": rpc error: code = NotFound desc = could not find container \"f0be24fdd8502dde12274400f774912fa9c3f1da236436e40d7046a6688fe37c\": container with ID starting with f0be24fdd8502dde12274400f774912fa9c3f1da236436e40d7046a6688fe37c not found: ID does not exist"
Jul 28 21:41:31.057799 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:31.057775    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"]
Jul 28 21:41:31.063095 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:31.063069    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-cf1484/ipp-route-cf1484-kserve-5cf78fd6dc-jhpk5"]
Jul 28 21:41:32.021243 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:32.021208    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1432ae88-40f7-40ac-aeb7-f84c243e6f9b" path="/var/lib/kubelet/pods/1432ae88-40f7-40ac-aeb7-f84c243e6f9b/volumes"
Jul 28 21:41:36.727120 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:36.727090    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:37.901878 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:37.901824    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:38.002895 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:38.002841    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:52.625844 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:52.625803    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:52.886505 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:52.886424    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:58.661234 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:58.661200    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:58.761918 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:58.761879    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:58.944434 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:58.944350    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:41:59.156304 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:41:59.156258    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:09.822725 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:09.822689    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:09.857910 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:09.857877    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"]
Jul 28 21:42:09.858200 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:09.858144    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn" podUID="c495140b-f5d2-4256-af03-bbd3db7b1e02" containerName="payload-pre-processing" containerID="cri-o://1b20ee9f2de88cb05036c55452be6baa3d2870859aeb4198109574b8b8f55c25" gracePeriod=30
Jul 28 21:42:09.923191 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:09.923155    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:09.996062 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:09.996035    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:10.062486 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.061796    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:10.121685 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.121656    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"
Jul 28 21:42:10.231315 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.231284    2537 generic.go:356] "Generic (PLEG): container finished" podID="c495140b-f5d2-4256-af03-bbd3db7b1e02" containerID="1b20ee9f2de88cb05036c55452be6baa3d2870859aeb4198109574b8b8f55c25" exitCode=0
Jul 28 21:42:10.231505 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.231335    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"
Jul 28 21:42:10.231505 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.231369    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn" event={"ID":"c495140b-f5d2-4256-af03-bbd3db7b1e02","Type":"ContainerDied","Data":"1b20ee9f2de88cb05036c55452be6baa3d2870859aeb4198109574b8b8f55c25"}
Jul 28 21:42:10.231505 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.231417    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn" event={"ID":"c495140b-f5d2-4256-af03-bbd3db7b1e02","Type":"ContainerDied","Data":"80ac1dee34ea01aa4d5a81348041890c1275737d6a5451444d2fb47dbd1c4036"}
Jul 28 21:42:10.231505 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.231440    2537 scope.go:117] "RemoveContainer" containerID="1b20ee9f2de88cb05036c55452be6baa3d2870859aeb4198109574b8b8f55c25"
Jul 28 21:42:10.237117 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.237090    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c495140b-f5d2-4256-af03-bbd3db7b1e02-plugins-config-volume\") pod \"c495140b-f5d2-4256-af03-bbd3db7b1e02\" (UID: \"c495140b-f5d2-4256-af03-bbd3db7b1e02\") "
Jul 28 21:42:10.237213 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.237136    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpjrb\" (UniqueName: \"kubernetes.io/projected/c495140b-f5d2-4256-af03-bbd3db7b1e02-kube-api-access-mpjrb\") pod \"c495140b-f5d2-4256-af03-bbd3db7b1e02\" (UID: \"c495140b-f5d2-4256-af03-bbd3db7b1e02\") "
Jul 28 21:42:10.237442 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.237424    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c495140b-f5d2-4256-af03-bbd3db7b1e02-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c495140b-f5d2-4256-af03-bbd3db7b1e02" (UID: "c495140b-f5d2-4256-af03-bbd3db7b1e02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:42:10.239230 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.239209    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c495140b-f5d2-4256-af03-bbd3db7b1e02-kube-api-access-mpjrb" (OuterVolumeSpecName: "kube-api-access-mpjrb") pod "c495140b-f5d2-4256-af03-bbd3db7b1e02" (UID: "c495140b-f5d2-4256-af03-bbd3db7b1e02"). InnerVolumeSpecName "kube-api-access-mpjrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:42:10.240268 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.240252    2537 scope.go:117] "RemoveContainer" containerID="1b20ee9f2de88cb05036c55452be6baa3d2870859aeb4198109574b8b8f55c25"
Jul 28 21:42:10.240532 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:42:10.240509    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b20ee9f2de88cb05036c55452be6baa3d2870859aeb4198109574b8b8f55c25\": container with ID starting with 1b20ee9f2de88cb05036c55452be6baa3d2870859aeb4198109574b8b8f55c25 not found: ID does not exist" containerID="1b20ee9f2de88cb05036c55452be6baa3d2870859aeb4198109574b8b8f55c25"
Jul 28 21:42:10.240593 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.240540    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b20ee9f2de88cb05036c55452be6baa3d2870859aeb4198109574b8b8f55c25"} err="failed to get container status \"1b20ee9f2de88cb05036c55452be6baa3d2870859aeb4198109574b8b8f55c25\": rpc error: code = NotFound desc = could not find container \"1b20ee9f2de88cb05036c55452be6baa3d2870859aeb4198109574b8b8f55c25\": container with ID starting with 1b20ee9f2de88cb05036c55452be6baa3d2870859aeb4198109574b8b8f55c25 not found: ID does not exist"
Jul 28 21:42:10.337991 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.337960    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c495140b-f5d2-4256-af03-bbd3db7b1e02-plugins-config-volume\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:10.337991 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.337988    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpjrb\" (UniqueName: \"kubernetes.io/projected/c495140b-f5d2-4256-af03-bbd3db7b1e02-kube-api-access-mpjrb\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:10.354671 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.354608    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:10.553893 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.553868    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"]
Jul 28 21:42:10.555689 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:10.555669    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d69596-574fd4f744-7cdtn"]
Jul 28 21:42:12.021659 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:12.021621    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c495140b-f5d2-4256-af03-bbd3db7b1e02" path="/var/lib/kubelet/pods/c495140b-f5d2-4256-af03-bbd3db7b1e02/volumes"
Jul 28 21:42:16.195842 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:16.195810    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:16.295522 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:16.295496    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:16.437696 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:16.437660    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"]
Jul 28 21:42:16.438014 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:16.437977    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz" podUID="606efe2f-25c8-4ad3-9fab-da5ad71f8b76" containerName="istio-proxy" containerID="cri-o://a89063e4f0879d660aa3a5d852160fb90eaa63720dbf0460c08681a98937b966" gracePeriod=30
Jul 28 21:42:16.452786 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:16.452724    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:16.692108 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:16.692078    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:20.131476 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:20.131450    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:42:20.144506 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:20.144483    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldz9r_de461c7c-9806-4aac-8afc-47e97696b0fb/ovn-acl-logging/0.log"
Jul 28 21:42:21.696923 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.696899    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:42:21.836630 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.836602    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-envoy\") pod \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") "
Jul 28 21:42:21.836813 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.836633    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-credential-socket\") pod \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") "
Jul 28 21:42:21.836813 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.836711    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-data\") pod \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") "
Jul 28 21:42:21.836813 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.836734    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwx7f\" (UniqueName: \"kubernetes.io/projected/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-kube-api-access-rwx7f\") pod \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") "
Jul 28 21:42:21.836813 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.836762    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-workload-socket\") pod \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") "
Jul 28 21:42:21.836813 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.836779    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istiod-ca-cert\") pod \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") "
Jul 28 21:42:21.836813 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.836798    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-podinfo\") pod \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") "
Jul 28 21:42:21.837187 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.836827    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-token\") pod \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") "
Jul 28 21:42:21.837187 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.836874    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-workload-certs\") pod \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\" (UID: \"606efe2f-25c8-4ad3-9fab-da5ad71f8b76\") "
Jul 28 21:42:21.837187 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.837082    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "606efe2f-25c8-4ad3-9fab-da5ad71f8b76" (UID: "606efe2f-25c8-4ad3-9fab-da5ad71f8b76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:42:21.837350 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.837192    2537 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-workload-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:21.837350 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.837203    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "606efe2f-25c8-4ad3-9fab-da5ad71f8b76" (UID: "606efe2f-25c8-4ad3-9fab-da5ad71f8b76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:42:21.837460 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.837358    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "606efe2f-25c8-4ad3-9fab-da5ad71f8b76" (UID: "606efe2f-25c8-4ad3-9fab-da5ad71f8b76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:42:21.837460 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.837439    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "606efe2f-25c8-4ad3-9fab-da5ad71f8b76" (UID: "606efe2f-25c8-4ad3-9fab-da5ad71f8b76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:42:21.837949 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.837920    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-data" (OuterVolumeSpecName: "istio-data") pod "606efe2f-25c8-4ad3-9fab-da5ad71f8b76" (UID: "606efe2f-25c8-4ad3-9fab-da5ad71f8b76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:42:21.839060 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.839030    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-kube-api-access-rwx7f" (OuterVolumeSpecName: "kube-api-access-rwx7f") pod "606efe2f-25c8-4ad3-9fab-da5ad71f8b76" (UID: "606efe2f-25c8-4ad3-9fab-da5ad71f8b76"). InnerVolumeSpecName "kube-api-access-rwx7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:42:21.839168 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.839025    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "606efe2f-25c8-4ad3-9fab-da5ad71f8b76" (UID: "606efe2f-25c8-4ad3-9fab-da5ad71f8b76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 21:42:21.839168 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.839084    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "606efe2f-25c8-4ad3-9fab-da5ad71f8b76" (UID: "606efe2f-25c8-4ad3-9fab-da5ad71f8b76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:42:21.839168 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.839089    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-token" (OuterVolumeSpecName: "istio-token") pod "606efe2f-25c8-4ad3-9fab-da5ad71f8b76" (UID: "606efe2f-25c8-4ad3-9fab-da5ad71f8b76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:42:21.938361 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.938333    2537 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-envoy\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:21.938361 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.938359    2537 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-credential-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:21.938534 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.938369    2537 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-data\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:21.938534 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.938377    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwx7f\" (UniqueName: \"kubernetes.io/projected/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-kube-api-access-rwx7f\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:21.938534 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.938387    2537 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istiod-ca-cert\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:21.938534 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.938395    2537 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-podinfo\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:21.938534 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.938402    2537 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-istio-token\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:21.938534 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:21.938410    2537 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606efe2f-25c8-4ad3-9fab-da5ad71f8b76-workload-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:22.293826 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:22.293737    2537 generic.go:356] "Generic (PLEG): container finished" podID="606efe2f-25c8-4ad3-9fab-da5ad71f8b76" containerID="a89063e4f0879d660aa3a5d852160fb90eaa63720dbf0460c08681a98937b966" exitCode=0
Jul 28 21:42:22.293826 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:22.293793    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"
Jul 28 21:42:22.294028 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:22.293821    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz" event={"ID":"606efe2f-25c8-4ad3-9fab-da5ad71f8b76","Type":"ContainerDied","Data":"a89063e4f0879d660aa3a5d852160fb90eaa63720dbf0460c08681a98937b966"}
Jul 28 21:42:22.294028 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:22.293903    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz" event={"ID":"606efe2f-25c8-4ad3-9fab-da5ad71f8b76","Type":"ContainerDied","Data":"d8c88e6ef0a0c9038dcb92c6ed414a528db55bbdfa610f25e73f1b769dde75da"}
Jul 28 21:42:22.294028 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:22.293929    2537 scope.go:117] "RemoveContainer" containerID="a89063e4f0879d660aa3a5d852160fb90eaa63720dbf0460c08681a98937b966"
Jul 28 21:42:22.303336 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:22.303316    2537 scope.go:117] "RemoveContainer" containerID="644a3c65da0c628c976d003afe04c84f64cd93c33aae48ebb27a8d096e0d0bf8"
Jul 28 21:42:22.311240 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:22.311216    2537 scope.go:117] "RemoveContainer" containerID="a89063e4f0879d660aa3a5d852160fb90eaa63720dbf0460c08681a98937b966"
Jul 28 21:42:22.311567 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:42:22.311543    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89063e4f0879d660aa3a5d852160fb90eaa63720dbf0460c08681a98937b966\": container with ID starting with a89063e4f0879d660aa3a5d852160fb90eaa63720dbf0460c08681a98937b966 not found: ID does not exist" containerID="a89063e4f0879d660aa3a5d852160fb90eaa63720dbf0460c08681a98937b966"
Jul 28 21:42:22.311679 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:22.311576    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89063e4f0879d660aa3a5d852160fb90eaa63720dbf0460c08681a98937b966"} err="failed to get container status \"a89063e4f0879d660aa3a5d852160fb90eaa63720dbf0460c08681a98937b966\": rpc error: code = NotFound desc = could not find container \"a89063e4f0879d660aa3a5d852160fb90eaa63720dbf0460c08681a98937b966\": container with ID starting with a89063e4f0879d660aa3a5d852160fb90eaa63720dbf0460c08681a98937b966 not found: ID does not exist"
Jul 28 21:42:22.311679 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:22.311599    2537 scope.go:117] "RemoveContainer" containerID="644a3c65da0c628c976d003afe04c84f64cd93c33aae48ebb27a8d096e0d0bf8"
Jul 28 21:42:22.311905 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:42:22.311884    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644a3c65da0c628c976d003afe04c84f64cd93c33aae48ebb27a8d096e0d0bf8\": container with ID starting with 644a3c65da0c628c976d003afe04c84f64cd93c33aae48ebb27a8d096e0d0bf8 not found: ID does not exist" containerID="644a3c65da0c628c976d003afe04c84f64cd93c33aae48ebb27a8d096e0d0bf8"
Jul 28 21:42:22.311994 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:22.311912    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644a3c65da0c628c976d003afe04c84f64cd93c33aae48ebb27a8d096e0d0bf8"} err="failed to get container status \"644a3c65da0c628c976d003afe04c84f64cd93c33aae48ebb27a8d096e0d0bf8\": rpc error: code = NotFound desc = could not find container \"644a3c65da0c628c976d003afe04c84f64cd93c33aae48ebb27a8d096e0d0bf8\": container with ID starting with 644a3c65da0c628c976d003afe04c84f64cd93c33aae48ebb27a8d096e0d0bf8 not found: ID does not exist"
Jul 28 21:42:22.313811 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:22.313789    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"]
Jul 28 21:42:22.317050 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:22.317029    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d69596-openshift-default-5b6d58f89d-nmltz"]
Jul 28 21:42:24.021634 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:24.021599    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="606efe2f-25c8-4ad3-9fab-da5ad71f8b76" path="/var/lib/kubelet/pods/606efe2f-25c8-4ad3-9fab-da5ad71f8b76/volumes"
Jul 28 21:42:27.406789 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:27.406758    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:27.473013 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:27.472974    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:27.530597 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:27.530557    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:27.592575 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:27.592546    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:27.952084 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:27.952056    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:34.051590 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:34.051538    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:34.152093 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:34.152057    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:34.285615 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:34.285581    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"]
Jul 28 21:42:34.285819 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:34.285797    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw" podUID="d51b6769-c17d-4e9e-85f8-fd0e819273f8" containerName="istio-proxy" containerID="cri-o://3e71a8f2262ad0e7371e3b55c35149e77f5ace797e38de49424931430bc344b6" gracePeriod=30
Jul 28 21:42:34.306081 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:34.305982    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:34.522144 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:34.522109    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:39.535545 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.535515    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:42:39.586825 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.586800    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-podinfo\") pod \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") "
Jul 28 21:42:39.586970 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.586842    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbxzf\" (UniqueName: \"kubernetes.io/projected/d51b6769-c17d-4e9e-85f8-fd0e819273f8-kube-api-access-cbxzf\") pod \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") "
Jul 28 21:42:39.586970 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.586935    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-workload-socket\") pod \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") "
Jul 28 21:42:39.586970 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.586966    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-envoy\") pod \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") "
Jul 28 21:42:39.587132 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.587000    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-workload-certs\") pod \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") "
Jul 28 21:42:39.587132 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.587041    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-token\") pod \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") "
Jul 28 21:42:39.587132 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.587085    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istiod-ca-cert\") pod \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") "
Jul 28 21:42:39.587132 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.587110    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-credential-socket\") pod \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") "
Jul 28 21:42:39.587349 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.587143    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-data\") pod \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\" (UID: \"d51b6769-c17d-4e9e-85f8-fd0e819273f8\") "
Jul 28 21:42:39.587349 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.587234    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d51b6769-c17d-4e9e-85f8-fd0e819273f8" (UID: "d51b6769-c17d-4e9e-85f8-fd0e819273f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:42:39.587489 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.587469    2537 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-workload-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:39.587765 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.587732    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d51b6769-c17d-4e9e-85f8-fd0e819273f8" (UID: "d51b6769-c17d-4e9e-85f8-fd0e819273f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:42:39.587950 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.587831    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d51b6769-c17d-4e9e-85f8-fd0e819273f8" (UID: "d51b6769-c17d-4e9e-85f8-fd0e819273f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:42:39.588174 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.588064    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d51b6769-c17d-4e9e-85f8-fd0e819273f8" (UID: "d51b6769-c17d-4e9e-85f8-fd0e819273f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:42:39.588555 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.588530    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "d51b6769-c17d-4e9e-85f8-fd0e819273f8" (UID: "d51b6769-c17d-4e9e-85f8-fd0e819273f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:42:39.589229 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.589210    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d51b6769-c17d-4e9e-85f8-fd0e819273f8-kube-api-access-cbxzf" (OuterVolumeSpecName: "kube-api-access-cbxzf") pod "d51b6769-c17d-4e9e-85f8-fd0e819273f8" (UID: "d51b6769-c17d-4e9e-85f8-fd0e819273f8"). InnerVolumeSpecName "kube-api-access-cbxzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:42:39.589333 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.589227    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d51b6769-c17d-4e9e-85f8-fd0e819273f8" (UID: "d51b6769-c17d-4e9e-85f8-fd0e819273f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:42:39.589397 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.589344    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d51b6769-c17d-4e9e-85f8-fd0e819273f8" (UID: "d51b6769-c17d-4e9e-85f8-fd0e819273f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 21:42:39.589737 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.589702    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "d51b6769-c17d-4e9e-85f8-fd0e819273f8" (UID: "d51b6769-c17d-4e9e-85f8-fd0e819273f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:42:39.688890 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.688836    2537 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-envoy\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:39.688890 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.688884    2537 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-workload-certs\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:39.688890 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.688894    2537 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-token\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:39.689126 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.688902    2537 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istiod-ca-cert\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:39.689126 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.688910    2537 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-credential-socket\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:39.689126 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.688919    2537 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-data\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:39.689126 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.688927    2537 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d51b6769-c17d-4e9e-85f8-fd0e819273f8-istio-podinfo\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:39.689126 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:39.688935    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbxzf\" (UniqueName: \"kubernetes.io/projected/d51b6769-c17d-4e9e-85f8-fd0e819273f8-kube-api-access-cbxzf\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:40.059403 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:42:40.059364    2537 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51b6769_c17d_4e9e_85f8_fd0e819273f8.slice/crio-03bbc76e748476023597381e3a11574e43ca2b01fc85b11160de9346e2e41e7b\": RecentStats: unable to find data in memory cache]"
Jul 28 21:42:40.385850 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:40.385817    2537 generic.go:356] "Generic (PLEG): container finished" podID="d51b6769-c17d-4e9e-85f8-fd0e819273f8" containerID="3e71a8f2262ad0e7371e3b55c35149e77f5ace797e38de49424931430bc344b6" exitCode=0
Jul 28 21:42:40.386059 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:40.385883    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"
Jul 28 21:42:40.386059 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:40.385901    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw" event={"ID":"d51b6769-c17d-4e9e-85f8-fd0e819273f8","Type":"ContainerDied","Data":"3e71a8f2262ad0e7371e3b55c35149e77f5ace797e38de49424931430bc344b6"}
Jul 28 21:42:40.386059 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:40.385957    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw" event={"ID":"d51b6769-c17d-4e9e-85f8-fd0e819273f8","Type":"ContainerDied","Data":"03bbc76e748476023597381e3a11574e43ca2b01fc85b11160de9346e2e41e7b"}
Jul 28 21:42:40.386059 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:40.385978    2537 scope.go:117] "RemoveContainer" containerID="3e71a8f2262ad0e7371e3b55c35149e77f5ace797e38de49424931430bc344b6"
Jul 28 21:42:40.394884 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:40.394849    2537 scope.go:117] "RemoveContainer" containerID="a1afa347d8143d966df99df767143add1e5c74d37811d61bf9f76f1613849ef7"
Jul 28 21:42:40.402662 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:40.402497    2537 scope.go:117] "RemoveContainer" containerID="3e71a8f2262ad0e7371e3b55c35149e77f5ace797e38de49424931430bc344b6"
Jul 28 21:42:40.402783 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:42:40.402765    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e71a8f2262ad0e7371e3b55c35149e77f5ace797e38de49424931430bc344b6\": container with ID starting with 3e71a8f2262ad0e7371e3b55c35149e77f5ace797e38de49424931430bc344b6 not found: ID does not exist" containerID="3e71a8f2262ad0e7371e3b55c35149e77f5ace797e38de49424931430bc344b6"
Jul 28 21:42:40.402832 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:40.402790    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e71a8f2262ad0e7371e3b55c35149e77f5ace797e38de49424931430bc344b6"} err="failed to get container status \"3e71a8f2262ad0e7371e3b55c35149e77f5ace797e38de49424931430bc344b6\": rpc error: code = NotFound desc = could not find container \"3e71a8f2262ad0e7371e3b55c35149e77f5ace797e38de49424931430bc344b6\": container with ID starting with 3e71a8f2262ad0e7371e3b55c35149e77f5ace797e38de49424931430bc344b6 not found: ID does not exist"
Jul 28 21:42:40.402832 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:40.402811    2537 scope.go:117] "RemoveContainer" containerID="a1afa347d8143d966df99df767143add1e5c74d37811d61bf9f76f1613849ef7"
Jul 28 21:42:40.403081 ip-10-0-133-14 kubenswrapper[2537]: E0728 21:42:40.403066    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1afa347d8143d966df99df767143add1e5c74d37811d61bf9f76f1613849ef7\": container with ID starting with a1afa347d8143d966df99df767143add1e5c74d37811d61bf9f76f1613849ef7 not found: ID does not exist" containerID="a1afa347d8143d966df99df767143add1e5c74d37811d61bf9f76f1613849ef7"
Jul 28 21:42:40.403134 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:40.403095    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1afa347d8143d966df99df767143add1e5c74d37811d61bf9f76f1613849ef7"} err="failed to get container status \"a1afa347d8143d966df99df767143add1e5c74d37811d61bf9f76f1613849ef7\": rpc error: code = NotFound desc = could not find container \"a1afa347d8143d966df99df767143add1e5c74d37811d61bf9f76f1613849ef7\": container with ID starting with a1afa347d8143d966df99df767143add1e5c74d37811d61bf9f76f1613849ef7 not found: ID does not exist"
Jul 28 21:42:40.410656 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:40.410635    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"]
Jul 28 21:42:40.412943 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:40.412922    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-1096ca-openshift-default-5d594c9476-pftlw"]
Jul 28 21:42:42.021400 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:42.021367    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d51b6769-c17d-4e9e-85f8-fd0e819273f8" path="/var/lib/kubelet/pods/d51b6769-c17d-4e9e-85f8-fd0e819273f8/volumes"
Jul 28 21:42:45.280839 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:45.280758    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"]
Jul 28 21:42:45.281278 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:45.281019    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6" podUID="39eebe04-47c4-412a-a685-e0a8528a5ea2" containerName="payload-processing" containerID="cri-o://a2c458ccc81b05efef294c2655e5399d67fe6a6d1a6bed9d760f81c59afbeb81" gracePeriod=30
Jul 28 21:42:45.414063 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:45.414035    2537 generic.go:356] "Generic (PLEG): container finished" podID="39eebe04-47c4-412a-a685-e0a8528a5ea2" containerID="a2c458ccc81b05efef294c2655e5399d67fe6a6d1a6bed9d760f81c59afbeb81" exitCode=0
Jul 28 21:42:45.414199 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:45.414110    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6" event={"ID":"39eebe04-47c4-412a-a685-e0a8528a5ea2","Type":"ContainerDied","Data":"a2c458ccc81b05efef294c2655e5399d67fe6a6d1a6bed9d760f81c59afbeb81"}
Jul 28 21:42:45.547584 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:45.547558    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"
Jul 28 21:42:45.637031 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:45.636999    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b5xr\" (UniqueName: \"kubernetes.io/projected/39eebe04-47c4-412a-a685-e0a8528a5ea2-kube-api-access-4b5xr\") pod \"39eebe04-47c4-412a-a685-e0a8528a5ea2\" (UID: \"39eebe04-47c4-412a-a685-e0a8528a5ea2\") "
Jul 28 21:42:45.637200 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:45.637052    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39eebe04-47c4-412a-a685-e0a8528a5ea2-plugins-config-volume\") pod \"39eebe04-47c4-412a-a685-e0a8528a5ea2\" (UID: \"39eebe04-47c4-412a-a685-e0a8528a5ea2\") "
Jul 28 21:42:45.637445 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:45.637422    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39eebe04-47c4-412a-a685-e0a8528a5ea2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39eebe04-47c4-412a-a685-e0a8528a5ea2" (UID: "39eebe04-47c4-412a-a685-e0a8528a5ea2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:42:45.639198 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:45.639177    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39eebe04-47c4-412a-a685-e0a8528a5ea2-kube-api-access-4b5xr" (OuterVolumeSpecName: "kube-api-access-4b5xr") pod "39eebe04-47c4-412a-a685-e0a8528a5ea2" (UID: "39eebe04-47c4-412a-a685-e0a8528a5ea2"). InnerVolumeSpecName "kube-api-access-4b5xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:42:45.737978 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:45.737943    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b5xr\" (UniqueName: \"kubernetes.io/projected/39eebe04-47c4-412a-a685-e0a8528a5ea2-kube-api-access-4b5xr\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:45.737978 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:45.737972    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39eebe04-47c4-412a-a685-e0a8528a5ea2-plugins-config-volume\") on node \"ip-10-0-133-14.ec2.internal\" DevicePath \"\""
Jul 28 21:42:45.907943 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:45.907906    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:46.008035 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:46.008005    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:46.420178 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:46.420147    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"
Jul 28 21:42:46.420672 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:46.420152    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6" event={"ID":"39eebe04-47c4-412a-a685-e0a8528a5ea2","Type":"ContainerDied","Data":"6c7b8123d832369d76def3c15f8eaf9e00685c4546372581ec71c44d3bd1757b"}
Jul 28 21:42:46.420672 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:46.420286    2537 scope.go:117] "RemoveContainer" containerID="a2c458ccc81b05efef294c2655e5399d67fe6a6d1a6bed9d760f81c59afbeb81"
Jul 28 21:42:46.438498 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:46.438474    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"]
Jul 28 21:42:46.442083 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:46.442057    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-2c3f70-55b8bdb9b6-vk2b6"]
Jul 28 21:42:48.020678 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:48.020649    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39eebe04-47c4-412a-a685-e0a8528a5ea2" path="/var/lib/kubelet/pods/39eebe04-47c4-412a-a685-e0a8528a5ea2/volumes"
Jul 28 21:42:51.973442 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:51.973409    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:52.074579 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:52.074544    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:52.221509 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:52.221477    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:52.438836 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:52.438804    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:42:52.855386 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:52.855356    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55cb49b747-rc952_cc8e82b2-950c-447a-8749-d5c097096f05/authorino/0.log"
Jul 28 21:42:57.974821 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:57.974794    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-t7qtj_42a32435-7136-4e80-b9ae-89576182daa4/manager/0.log"
Jul 28 21:42:58.083844 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:58.083810    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-n4vrg_8768f71c-7578-4d35-bab3-66352dea99d8/manager/0.log"
Jul 28 21:42:58.195241 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:58.195209    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8555f666d8-9vdt2_d1ec31bc-c698-416c-9bd4-f48888f77a9c/manager/0.log"
Jul 28 21:42:58.303604 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:58.303514    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-kxjcl_3b782743-5f3e-4c6b-abfd-3b726e6841cc/server/0.log"
Jul 28 21:42:58.805023 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:58.804995    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cc774576c-qt8l9_bcfc884a-fe66-4921-9bec-6d8c8723cf4a/copy-manifests/0.log"
Jul 28 21:42:58.811321 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:58.811298    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cc774576c-qt8l9_bcfc884a-fe66-4921-9bec-6d8c8723cf4a/manager/0.log"
Jul 28 21:42:59.135999 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:59.135969    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6cd98d97c9-9npjb_00544e1f-fb7f-413b-8cf1-549e66a1e3af/maas-api/0.log"
Jul 28 21:42:59.568449 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:42:59.568425    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-d7578_169e9d37-e577-4211-a26b-ef367e1d06b6/postgres/0.log"
Jul 28 21:43:00.826646 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:00.826617    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55cb49b747-rc952_cc8e82b2-950c-447a-8749-d5c097096f05/authorino/0.log"
Jul 28 21:43:00.941328 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:00.941285    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-82sks_959b3684-e080-494d-aefd-34c648b9c66e/manager/0.log"
Jul 28 21:43:01.045397 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:01.045370    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-9pxl8_968405d0-4f21-454c-9c39-2367334686a8/manager/0.log"
Jul 28 21:43:01.272028 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:01.271941    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-82mc7_c39ea625-8476-4119-8960-6cda4db3bce2/registry-server/0.log"
Jul 28 21:43:01.610628 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:01.610591    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hh2m8_99de2739-1bad-4816-8cc0-031dfab10755/manager/0.log"
Jul 28 21:43:02.275053 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:02.275022    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68cb4f9b6-pmtxl_618aeb6b-d661-425a-b379-04280daef097/kube-auth-proxy/0.log"
Jul 28 21:43:02.389981 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:02.389955    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8qz9v_9b268625-85ea-4240-a31f-15047c283421/istio-proxy/1.log"
Jul 28 21:43:02.501445 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:02.501415    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-jh9xk_56ec6091-a220-464c-9f02-02f631a10e3e/payload-pre-processing/0.log"
Jul 28 21:43:02.623440 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:02.623414    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-zjcmp_32df0cb1-de8b-4183-a6c5-9e0e3c03a67e/payload-processing/0.log"
Jul 28 21:43:02.735350 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:02.735315    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d8755fdbc-w6k7x_8260b33a-e496-4638-996c-7f25da54964f/router/0.log"
Jul 28 21:43:03.069217 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:03.069190    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-g4dg2_7e45bef1-f7a2-47b0-b08e-fde66c0302b7/main/0.log"
Jul 28 21:43:03.288818 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:03.288792    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-zw679_5e54d3b1-49cc-4160-ab1a-50a13ff59e9b/main/0.log"
Jul 28 21:43:03.614246 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:03.614220    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-zbv8q_16133060-8272-4e80-8877-f640a1b55eb4/main/0.log"
Jul 28 21:43:08.651099 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:08.651066    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:43:10.313203 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:10.313173    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:43:11.529114 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:11.529076    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:43:11.590199 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:11.590176    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:43:11.739765 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:11.739738    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xx9kc_415a6e79-19ba-4beb-ab92-8b2b739d63a4/global-pull-secret-syncer/0.log"
Jul 28 21:43:11.802389 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:11.802314    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log"
Jul 28 21:43:11.913557 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:11.913531    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-14.ec2.internal_055a7a05e8fd2212ddcaf08826be7843/haproxy/0.log"
Jul 28 21:43:17.036515 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:17.036478    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55cb49b747-rc952_cc8e82b2-950c-447a-8749-d5c097096f05/authorino/0.log"
Jul 28 21:43:17.074018 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:17.073992    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-82sks_959b3684-e080-494d-aefd-34c648b9c66e/manager/0.log"
Jul 28 21:43:17.106433 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:17.106414    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-9pxl8_968405d0-4f21-454c-9c39-2367334686a8/manager/0.log"
Jul 28 21:43:17.209649 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:17.209623    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-82mc7_c39ea625-8476-4119-8960-6cda4db3bce2/registry-server/0.log"
Jul 28 21:43:17.582368 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:17.582337    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hh2m8_99de2739-1bad-4816-8cc0-031dfab10755/manager/0.log"
Jul 28 21:43:19.590227 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:19.590196    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-qspqh_96b558af-774c-4a05-a70f-0ebe768ebeee/kube-state-metrics/0.log"
Jul 28 21:43:19.612568 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:19.612541    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-qspqh_96b558af-774c-4a05-a70f-0ebe768ebeee/kube-rbac-proxy-main/0.log"
Jul 28 21:43:19.635881 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:19.635845    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-qspqh_96b558af-774c-4a05-a70f-0ebe768ebeee/kube-rbac-proxy-self/0.log"
Jul 28 21:43:19.692394 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:19.692367    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-4bjnc_3d268dea-f385-4c8d-9a65-ca2c8e9d6ff3/monitoring-plugin/0.log"
Jul 28 21:43:19.724273 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:19.724243    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4nmrq_303a511a-4ac0-4b7d-bc70-e21fde77d433/node-exporter/0.log"
Jul 28 21:43:19.757746 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:19.757715    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4nmrq_303a511a-4ac0-4b7d-bc70-e21fde77d433/kube-rbac-proxy/0.log"
Jul 28 21:43:19.781407 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:19.781383    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4nmrq_303a511a-4ac0-4b7d-bc70-e21fde77d433/init-textfile/0.log"
Jul 28 21:43:21.848809 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.848774    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"]
Jul 28 21:43:21.849221 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849197    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="606efe2f-25c8-4ad3-9fab-da5ad71f8b76" containerName="istio-proxy"
Jul 28 21:43:21.849221 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849209    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="606efe2f-25c8-4ad3-9fab-da5ad71f8b76" containerName="istio-proxy"
Jul 28 21:43:21.849221 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849217    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d51b6769-c17d-4e9e-85f8-fd0e819273f8" containerName="istio-proxy"
Jul 28 21:43:21.849221 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849223    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="d51b6769-c17d-4e9e-85f8-fd0e819273f8" containerName="istio-proxy"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849229    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="606efe2f-25c8-4ad3-9fab-da5ad71f8b76" containerName="istio-proxy"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849236    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="606efe2f-25c8-4ad3-9fab-da5ad71f8b76" containerName="istio-proxy"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849243    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71389d21-4bdc-4da0-8592-4ea8e3a7f4e3" containerName="payload-pre-processing"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849248    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="71389d21-4bdc-4da0-8592-4ea8e3a7f4e3" containerName="payload-pre-processing"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849256    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="269e8a5c-e26c-4f24-bd40-1588baa67828" containerName="payload-processing"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849262    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="269e8a5c-e26c-4f24-bd40-1588baa67828" containerName="payload-processing"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849285    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1432ae88-40f7-40ac-aeb7-f84c243e6f9b" containerName="main"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849290    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="1432ae88-40f7-40ac-aeb7-f84c243e6f9b" containerName="main"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849297    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c495140b-f5d2-4256-af03-bbd3db7b1e02" containerName="payload-pre-processing"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849302    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="c495140b-f5d2-4256-af03-bbd3db7b1e02" containerName="payload-pre-processing"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849316    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39eebe04-47c4-412a-a685-e0a8528a5ea2" containerName="payload-processing"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849323    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="39eebe04-47c4-412a-a685-e0a8528a5ea2" containerName="payload-processing"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849328    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04abd142-5c8a-4567-966f-a41612901d82" containerName="payload-pre-processing"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849333    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="04abd142-5c8a-4567-966f-a41612901d82" containerName="payload-pre-processing"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849342    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d51b6769-c17d-4e9e-85f8-fd0e819273f8" containerName="istio-proxy"
Jul 28 21:43:21.849363 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849347    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="d51b6769-c17d-4e9e-85f8-fd0e819273f8" containerName="istio-proxy"
Jul 28 21:43:21.849835 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849398    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="1432ae88-40f7-40ac-aeb7-f84c243e6f9b" containerName="main"
Jul 28 21:43:21.849835 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849407    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="269e8a5c-e26c-4f24-bd40-1588baa67828" containerName="payload-processing"
Jul 28 21:43:21.849835 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849413    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="d51b6769-c17d-4e9e-85f8-fd0e819273f8" containerName="istio-proxy"
Jul 28 21:43:21.849835 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849419    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="04abd142-5c8a-4567-966f-a41612901d82" containerName="payload-pre-processing"
Jul 28 21:43:21.849835 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849424    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="d51b6769-c17d-4e9e-85f8-fd0e819273f8" containerName="istio-proxy"
Jul 28 21:43:21.849835 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849430    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="c495140b-f5d2-4256-af03-bbd3db7b1e02" containerName="payload-pre-processing"
Jul 28 21:43:21.849835 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849435    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="39eebe04-47c4-412a-a685-e0a8528a5ea2" containerName="payload-processing"
Jul 28 21:43:21.849835 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849440    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="606efe2f-25c8-4ad3-9fab-da5ad71f8b76" containerName="istio-proxy"
Jul 28 21:43:21.849835 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.849447    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="71389d21-4bdc-4da0-8592-4ea8e3a7f4e3" containerName="payload-pre-processing"
Jul 28 21:43:21.853918 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.853896    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:21.856384 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.856352    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grfnp\"/\"kube-root-ca.crt\""
Jul 28 21:43:21.856497 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.856369    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grfnp\"/\"openshift-service-ca.crt\""
Jul 28 21:43:21.856915 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.856900    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-grfnp\"/\"default-dockercfg-trz77\""
Jul 28 21:43:21.860630 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.860609    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"]
Jul 28 21:43:21.946237 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.946210    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/026c88cd-5174-414e-92c7-8c2a1f54f7f8-proc\") pod \"perf-node-gather-daemonset-w7dnh\" (UID: \"026c88cd-5174-414e-92c7-8c2a1f54f7f8\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:21.946390 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.946257    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/026c88cd-5174-414e-92c7-8c2a1f54f7f8-podres\") pod \"perf-node-gather-daemonset-w7dnh\" (UID: \"026c88cd-5174-414e-92c7-8c2a1f54f7f8\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:21.946390 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.946297    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026c88cd-5174-414e-92c7-8c2a1f54f7f8-sys\") pod \"perf-node-gather-daemonset-w7dnh\" (UID: \"026c88cd-5174-414e-92c7-8c2a1f54f7f8\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:21.946390 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.946353    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/026c88cd-5174-414e-92c7-8c2a1f54f7f8-lib-modules\") pod \"perf-node-gather-daemonset-w7dnh\" (UID: \"026c88cd-5174-414e-92c7-8c2a1f54f7f8\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:21.946390 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:21.946371    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh8qn\" (UniqueName: \"kubernetes.io/projected/026c88cd-5174-414e-92c7-8c2a1f54f7f8-kube-api-access-sh8qn\") pod \"perf-node-gather-daemonset-w7dnh\" (UID: \"026c88cd-5174-414e-92c7-8c2a1f54f7f8\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:22.047346 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:22.047313    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/026c88cd-5174-414e-92c7-8c2a1f54f7f8-lib-modules\") pod \"perf-node-gather-daemonset-w7dnh\" (UID: \"026c88cd-5174-414e-92c7-8c2a1f54f7f8\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:22.047346 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:22.047346    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh8qn\" (UniqueName: \"kubernetes.io/projected/026c88cd-5174-414e-92c7-8c2a1f54f7f8-kube-api-access-sh8qn\") pod \"perf-node-gather-daemonset-w7dnh\" (UID: \"026c88cd-5174-414e-92c7-8c2a1f54f7f8\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:22.047634 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:22.047378    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/026c88cd-5174-414e-92c7-8c2a1f54f7f8-proc\") pod \"perf-node-gather-daemonset-w7dnh\" (UID: \"026c88cd-5174-414e-92c7-8c2a1f54f7f8\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:22.047634 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:22.047441    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/026c88cd-5174-414e-92c7-8c2a1f54f7f8-proc\") pod \"perf-node-gather-daemonset-w7dnh\" (UID: \"026c88cd-5174-414e-92c7-8c2a1f54f7f8\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:22.047634 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:22.047490    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/026c88cd-5174-414e-92c7-8c2a1f54f7f8-podres\") pod \"perf-node-gather-daemonset-w7dnh\" (UID: \"026c88cd-5174-414e-92c7-8c2a1f54f7f8\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:22.047634 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:22.047512    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/026c88cd-5174-414e-92c7-8c2a1f54f7f8-lib-modules\") pod \"perf-node-gather-daemonset-w7dnh\" (UID: \"026c88cd-5174-414e-92c7-8c2a1f54f7f8\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:22.047634 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:22.047543    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026c88cd-5174-414e-92c7-8c2a1f54f7f8-sys\") pod \"perf-node-gather-daemonset-w7dnh\" (UID: \"026c88cd-5174-414e-92c7-8c2a1f54f7f8\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:22.047634 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:22.047605    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/026c88cd-5174-414e-92c7-8c2a1f54f7f8-podres\") pod \"perf-node-gather-daemonset-w7dnh\" (UID: \"026c88cd-5174-414e-92c7-8c2a1f54f7f8\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:22.047634 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:22.047637    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026c88cd-5174-414e-92c7-8c2a1f54f7f8-sys\") pod \"perf-node-gather-daemonset-w7dnh\" (UID: \"026c88cd-5174-414e-92c7-8c2a1f54f7f8\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:22.055976 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:22.055958    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh8qn\" (UniqueName: \"kubernetes.io/projected/026c88cd-5174-414e-92c7-8c2a1f54f7f8-kube-api-access-sh8qn\") pod \"perf-node-gather-daemonset-w7dnh\" (UID: \"026c88cd-5174-414e-92c7-8c2a1f54f7f8\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:22.165083 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:22.165012    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:22.506991 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:22.506967    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"]
Jul 28 21:43:22.509052 ip-10-0-133-14 kubenswrapper[2537]: W0728 21:43:22.509025    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod026c88cd_5174_414e_92c7_8c2a1f54f7f8.slice/crio-2ce2fc276c9ca5c5fcfa269a7d0061bd90546c5f26ca260bcea0ecb4b20da96e WatchSource:0}: Error finding container 2ce2fc276c9ca5c5fcfa269a7d0061bd90546c5f26ca260bcea0ecb4b20da96e: Status 404 returned error can't find the container with id 2ce2fc276c9ca5c5fcfa269a7d0061bd90546c5f26ca260bcea0ecb4b20da96e
Jul 28 21:43:22.593962 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:22.593929    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh" event={"ID":"026c88cd-5174-414e-92c7-8c2a1f54f7f8","Type":"ContainerStarted","Data":"2ce2fc276c9ca5c5fcfa269a7d0061bd90546c5f26ca260bcea0ecb4b20da96e"}
Jul 28 21:43:23.458013 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:23.457973    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-5p5wl_41fbe0e3-2435-4381-940e-508116343854/volume-data-source-validator/0.log"
Jul 28 21:43:23.600395 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:23.600357    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh" event={"ID":"026c88cd-5174-414e-92c7-8c2a1f54f7f8","Type":"ContainerStarted","Data":"7ef7a1d418bf4cfe7c1f3c0693002088727e1c5425ede36770de712563ca9dd5"}
Jul 28 21:43:23.600562 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:23.600421    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:23.616325 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:23.616275    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh" podStartSLOduration=2.616260482 podStartE2EDuration="2.616260482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 21:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:43:23.615651839 +0000 UTC m=+3364.233233752" watchObservedRunningTime="2026-07-28 21:43:23.616260482 +0000 UTC m=+3364.233842386"
Jul 28 21:43:24.486636 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:24.486607    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mrnqh_0b960510-a109-42a8-99d8-dad53a778237/dns/0.log"
Jul 28 21:43:24.509630 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:24.509603    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mrnqh_0b960510-a109-42a8-99d8-dad53a778237/kube-rbac-proxy/0.log"
Jul 28 21:43:24.692208 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:24.692172    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tk54b_4421ad59-d310-4fce-b6f2-5518ede532fa/dns-node-resolver/0.log"
Jul 28 21:43:25.306721 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:25.306694    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6vhdp_19ba6b7f-59fc-4288-aef0-a479c989e6d7/node-ca/0.log"
Jul 28 21:43:26.587997 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:26.587963    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68cb4f9b6-pmtxl_618aeb6b-d661-425a-b379-04280daef097/kube-auth-proxy/0.log"
Jul 28 21:43:26.621240 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:26.621212    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8qz9v_9b268625-85ea-4240-a31f-15047c283421/istio-proxy/0.log"
Jul 28 21:43:26.646908 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:26.646881    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8qz9v_9b268625-85ea-4240-a31f-15047c283421/istio-proxy/1.log"
Jul 28 21:43:26.678078 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:26.678055    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-jh9xk_56ec6091-a220-464c-9f02-02f631a10e3e/payload-pre-processing/0.log"
Jul 28 21:43:26.708573 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:26.708549    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-zjcmp_32df0cb1-de8b-4183-a6c5-9e0e3c03a67e/payload-processing/0.log"
Jul 28 21:43:26.730679 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:26.730659    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d8755fdbc-w6k7x_8260b33a-e496-4638-996c-7f25da54964f/router/0.log"
Jul 28 21:43:27.631942 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:27.631900    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r2g2c_8083d69e-72cb-48d1-91bb-286ee0b4141d/serve-healthcheck-canary/0.log"
Jul 28 21:43:28.116821 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:28.116794    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-mpflg_5cd1b546-a068-41fc-9b32-ff5c2295a654/insights-operator/0.log"
Jul 28 21:43:28.118383 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:28.118363    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-mpflg_5cd1b546-a068-41fc-9b32-ff5c2295a654/insights-operator/1.log"
Jul 28 21:43:28.225537 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:28.225514    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gprrj_55e48be1-28c7-409d-baf5-939aeeff7d34/kube-rbac-proxy/0.log"
Jul 28 21:43:28.250250 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:28.250228    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gprrj_55e48be1-28c7-409d-baf5-939aeeff7d34/exporter/0.log"
Jul 28 21:43:28.287632 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:28.287614    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gprrj_55e48be1-28c7-409d-baf5-939aeeff7d34/extractor/0.log"
Jul 28 21:43:29.615114 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:29.615086    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-w7dnh"
Jul 28 21:43:30.563252 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:30.563222    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-t7qtj_42a32435-7136-4e80-b9ae-89576182daa4/manager/0.log"
Jul 28 21:43:30.788588 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:30.788563    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-n4vrg_8768f71c-7578-4d35-bab3-66352dea99d8/manager/0.log"
Jul 28 21:43:30.928534 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:30.928509    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8555f666d8-9vdt2_d1ec31bc-c698-416c-9bd4-f48888f77a9c/manager/0.log"
Jul 28 21:43:30.952294 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:30.952270    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-kxjcl_3b782743-5f3e-4c6b-abfd-3b726e6841cc/server/0.log"
Jul 28 21:43:31.314326 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:31.314301    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cc774576c-qt8l9_bcfc884a-fe66-4921-9bec-6d8c8723cf4a/manager/0.log"
Jul 28 21:43:31.339288 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:31.339261    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cc774576c-qt8l9_bcfc884a-fe66-4921-9bec-6d8c8723cf4a/copy-manifests/0.log"
Jul 28 21:43:32.507703 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:32.507675    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68cb4f9b6-pmtxl_618aeb6b-d661-425a-b379-04280daef097/kube-auth-proxy/0.log"
Jul 28 21:43:32.525841 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:32.525813    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8qz9v_9b268625-85ea-4240-a31f-15047c283421/istio-proxy/0.log"
Jul 28 21:43:32.551354 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:32.551329    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8qz9v_9b268625-85ea-4240-a31f-15047c283421/istio-proxy/1.log"
Jul 28 21:43:32.575372 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:32.575350    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-jh9xk_56ec6091-a220-464c-9f02-02f631a10e3e/payload-pre-processing/0.log"
Jul 28 21:43:32.592968 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:32.592949    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-zjcmp_32df0cb1-de8b-4183-a6c5-9e0e3c03a67e/payload-processing/0.log"
Jul 28 21:43:32.608684 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:32.608668    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d8755fdbc-w6k7x_8260b33a-e496-4638-996c-7f25da54964f/router/0.log"
Jul 28 21:43:34.360708 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:34.360680    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68cb4f9b6-pmtxl_618aeb6b-d661-425a-b379-04280daef097/kube-auth-proxy/0.log"
Jul 28 21:43:34.379442 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:34.379419    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8qz9v_9b268625-85ea-4240-a31f-15047c283421/istio-proxy/0.log"
Jul 28 21:43:34.403665 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:34.403642    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8qz9v_9b268625-85ea-4240-a31f-15047c283421/istio-proxy/1.log"
Jul 28 21:43:34.426061 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:34.426043    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-jh9xk_56ec6091-a220-464c-9f02-02f631a10e3e/payload-pre-processing/0.log"
Jul 28 21:43:34.446170 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:34.446151    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-zjcmp_32df0cb1-de8b-4183-a6c5-9e0e3c03a67e/payload-processing/0.log"
Jul 28 21:43:34.461310 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:34.461292    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d8755fdbc-w6k7x_8260b33a-e496-4638-996c-7f25da54964f/router/0.log"
Jul 28 21:43:38.443161 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:38.443122    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44fv_65bbeae6-101e-450a-bdcc-6d3c9684dad5/konnectivity-agent/0.log" limit=1024
Jul 28 21:43:39.231607 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:39.231576    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-drhrl_bb0b5897-6159-4391-a86a-107312cea21e/kube-multus-additional-cni-plugins/0.log"
Jul 28 21:43:39.258094 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:39.258067    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-drhrl_bb0b5897-6159-4391-a86a-107312cea21e/egress-router-binary-copy/0.log"
Jul 28 21:43:39.284255 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:39.284235    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-drhrl_bb0b5897-6159-4391-a86a-107312cea21e/cni-plugins/0.log"
Jul 28 21:43:39.308161 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:39.308144    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-drhrl_bb0b5897-6159-4391-a86a-107312cea21e/bond-cni-plugin/0.log"
Jul 28 21:43:39.337203 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:39.337182    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-drhrl_bb0b5897-6159-4391-a86a-107312cea21e/routeoverride-cni/0.log"
Jul 28 21:43:39.377768 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:39.377748    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-drhrl_bb0b5897-6159-4391-a86a-107312cea21e/whereabouts-cni-bincopy/0.log"
Jul 28 21:43:39.418027 ip-10-0-133-14 kubenswrapper[2537]: I0728 21:43:39.418003    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-drhrl_bb0b5897-6159-4391-a86a-107312cea21e/whereabouts-cni/0.log"