Jul 14 17:30:11.405210 ip-10-0-135-231 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 14 17:30:11.405222 ip-10-0-135-231 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 14 17:30:11.405231 ip-10-0-135-231 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 14 17:30:11.405539 ip-10-0-135-231 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 14 17:30:22.753204 ip-10-0-135-231 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 14 17:30:22.753221 ip-10-0-135-231 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7c9fad9db63344dcb525816b211dc5d1 --
Jul 14 17:32:23.164500 ip-10-0-135-231 systemd[1]: Starting Kubernetes Kubelet...
Jul 14 17:32:23.545916 ip-10-0-135-231 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 17:32:23.545916 ip-10-0-135-231 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 14 17:32:23.545916 ip-10-0-135-231 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 17:32:23.545916 ip-10-0-135-231 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 14 17:32:23.545916 ip-10-0-135-231 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 17:32:23.548316 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.548224    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 14 17:32:23.553283 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553265    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 17:32:23.553283 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553283    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553288    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553291    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553296    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553298    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553301    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553304    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553306    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553309    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553311    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553314    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553316    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553319    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553322    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553325    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553327    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553330    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553332    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553342    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553345    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 17:32:23.553353 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553348    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553351    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553353    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553356    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553359    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553362    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553364    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553367    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553369    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553372    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553374    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553377    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553379    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553382    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553384    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553386    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553389    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553391    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553394    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553396    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 17:32:23.553816 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553399    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553402    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553405    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553407    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553409    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553412    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553414    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553417    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553420    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553423    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553426    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553428    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553430    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553433    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553436    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553439    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553442    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553445    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553447    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553449    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 17:32:23.554315 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553452    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553454    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553457    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553459    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553462    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553464    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553467    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553470    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553474    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553478    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553482    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553485    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553488    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553493    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553497    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553500    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553503    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553505    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 17:32:23.554800 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553508    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553510    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553512    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553515    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553517    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553520    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553522    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553526    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553529    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553531    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553534    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553539    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553542    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553545    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553548    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553551    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553553    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553556    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553558    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.553561    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 17:32:23.555269 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554304    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554312    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554316    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554319    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554322    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554325    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554329    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554332    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554334    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554337    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554340    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554342    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554345    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554347    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554350    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554352    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554355    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554357    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554360    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 17:32:23.555752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554363    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554365    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554368    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554371    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554374    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554376    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554378    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554381    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554383    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554386    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554388    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554390    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554393    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554395    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554397    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554400    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554403    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554405    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554408    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554411    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 17:32:23.556270 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554415    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554417    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554420    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554422    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554425    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554427    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554429    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554432    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554434    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554437    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554439    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554441    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554444    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554447    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554449    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554452    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554454    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554456    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554460    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 17:32:23.556752 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554465    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554468    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554471    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554473    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554476    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554479    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554482    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554484    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554487    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554490    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554492    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554494    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554497    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554500    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554505    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554508    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554511    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554514    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554516    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554518    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 17:32:23.557235 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554521    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554523    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554526    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554528    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554531    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554533    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554537    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554539    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554542    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554544    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554547    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554550    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554552    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554554    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554557    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554559    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554562    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554564    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554566    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554569    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 17:32:23.557728 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554571    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554931    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554936    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554940    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554943    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554947    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554950    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554954    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554956    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554959    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554962    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554965    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554967    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554970    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554972    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554975    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554978    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554981    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554983    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 17:32:23.558242 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554986    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554989    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554991    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554994    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554997    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.554999    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555002    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555004    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555006    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555009    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555011    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555014    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555016    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555019    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555022    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555024    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555027    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555029    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555032    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555035    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 17:32:23.558699 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555038    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555041    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555043    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555045    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555051    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555053    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555055    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555058    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555061    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555063    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555066    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555069    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555071    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555074    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555077    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555079    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555082    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555084    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555087    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 17:32:23.559176 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555089    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555092    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555094    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555097    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555099    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555102    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555105    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555107    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555110    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555112    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555115    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555118    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555121    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555123    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555126    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555128    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555131    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555135    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555138    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555140    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 17:32:23.559649 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555143    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555145    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555148    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555151    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555154    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555157    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555161    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555164    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555166    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555169    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555172    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555175    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555178    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555181    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555183    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555186    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555189    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555192    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555194    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555197    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 17:32:23.560124 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555200    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555218    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555298    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555309    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555318    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555324    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555331    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555334    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555339    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555345    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555348    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555351    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555354    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555358    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555361    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555364    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555367    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555370    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555373    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555376    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555380    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555383    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555386    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555389    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 14 17:32:23.560597 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555392    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555396    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555399    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555402    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555405    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555408    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555411    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555414    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555417    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555419    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555424    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555427    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555430    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555433    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555436    2578 flags.go:64] FLAG: --enable-server="true"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555439    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555443    2578 flags.go:64] FLAG: --event-burst="100"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555447    2578 flags.go:64] FLAG: --event-qps="50"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555450    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555453    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555456    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555460    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555464    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555467    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555469    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 14 17:32:23.561200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555472    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555475    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555478    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555480    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555483    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555486    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555489    2578 flags.go:64] FLAG: --feature-gates=""
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555492    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555495    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555498    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555501    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555504    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555507    2578 flags.go:64] FLAG: --help="false"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555511    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-135-231.ec2.internal"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555513    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555516    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555519    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555522    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555525    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555528    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555531    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555533    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555536    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 14 17:32:23.561805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555540    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555544    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555547    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555550    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555553    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555555    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555560    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555563    2578 flags.go:64] FLAG: --lock-file=""
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555566    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555568    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555572    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555580    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555591    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555595    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555598    2578 flags.go:64] FLAG: --logging-format="text"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555601    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555604    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555607    2578 flags.go:64] FLAG: --manifest-url=""
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555610    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555614    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555617    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555621    2578 flags.go:64] FLAG: --max-pods="110"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555624    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555627    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555629    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 14 17:32:23.562405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555632    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555635    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555638    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555641    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555648    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555651    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555654    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555656    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555661    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555667    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555670    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555673    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555675    2578 flags.go:64] FLAG: --port="10250"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555680    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555682    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-009a0c8f37565f425"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555685    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555688    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555691    2578 flags.go:64] FLAG: --register-node="true"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555694    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555697    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555700    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555703    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555706    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555709    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555712    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 14 17:32:23.563007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555715    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555718    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555720    2578 flags.go:64] FLAG: --runonce="false"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555723    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555726    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555729    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555732    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555734    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555737    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555740    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555743    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555746    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555748    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555751    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555754    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555757    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555761    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555764    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555770    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555772    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555776    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555781    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555783    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555786    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555789    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 14 17:32:23.563664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555792    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555795    2578 flags.go:64] FLAG: --v="2"
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555799    2578 flags.go:64] FLAG: --version="false"
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555802    2578 flags.go:64] FLAG: --vmodule=""
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555806    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.555809    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555923    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555927    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555930    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555933    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555935    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555939    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555941    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555943    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555946    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555948    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555951    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555954    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555956    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555959    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555961    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 17:32:23.564286 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555963    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555966    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555969    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555973    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555975    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555977    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555981    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555983    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555986    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555988    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555991    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555993    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555996    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.555998    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556001    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556003    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556006    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556008    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556011    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 17:32:23.564840 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556013    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556016    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556018    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556020    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556023    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556025    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556027    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556030    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556032    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556035    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556039    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556042    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556046    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556048    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556051    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556055    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556059    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556063    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556066    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 17:32:23.565310 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556070    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556073    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556076    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556079    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556081    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556084    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556087    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556089    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556092    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556094    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556097    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556099    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556103    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556107    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556109    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556112    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556114    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556117    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556119    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556121    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 17:32:23.565831 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556124    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556126    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556129    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556131    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556133    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556136    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556139    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556141    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556143    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556146    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556149    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556151    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556155    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556158    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556160    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556163    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556165    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556168    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556170    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 17:32:23.566445 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556172    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 17:32:23.566994 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556175    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 17:32:23.566994 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556177    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 17:32:23.566994 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556180    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 17:32:23.566994 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556183    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 17:32:23.566994 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556185    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 17:32:23.566994 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.556188    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 17:32:23.566994 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.556697    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 17:32:23.566994 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.563626    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 14 17:32:23.566994 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.563643    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 14 17:32:23.566994 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.563664    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 14 17:32:23.566994 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.563671    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 14 17:32:23.566994 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563726    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 17:32:23.566994 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563731    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 17:32:23.566994 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563735    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 17:32:23.566994 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563738    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563741    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563744    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563747    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563750    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563753    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563756    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563758    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563761    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563764    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563767    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563770    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563774    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563778    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563781    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563784    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563786    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563789    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563792    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 17:32:23.567397 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563794    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563797    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563799    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563802    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563804    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563807    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563810    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563814    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563817    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563819    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563822    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563824    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563827    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563829    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563833    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563836    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563839    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563843    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563845    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563848    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 17:32:23.567955 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563851    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563854    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563857    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563859    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563862    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563864    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563867    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563869    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563873    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563877    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563880    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563883    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563885    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563888    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563891    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563894    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563896    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563899    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563901    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 17:32:23.568505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563904    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563907    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563910    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563913    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563916    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563919    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563921    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563924    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563926    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563928    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563931    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563933    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563936    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563939    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563941    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563943    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563946    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563948    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563951    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563953    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 17:32:23.568973 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563956    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563958    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563961    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563964    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563966    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563969    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563971    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563974    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563976    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563979    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563981    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563984    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563987    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563990    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563993    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563995    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.563998    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 17:32:23.569477 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564000    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 17:32:23.570096 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.564005    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 17:32:23.570096 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564121    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 17:32:23.570096 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564125    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 17:32:23.570096 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564128    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 17:32:23.570096 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564131    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 17:32:23.570096 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564134    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 17:32:23.570096 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564136    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 17:32:23.570096 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564139    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 17:32:23.570096 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564141    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 17:32:23.570096 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564144    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 17:32:23.570096 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564147    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 17:32:23.570096 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564149    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 17:32:23.570096 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564153    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 17:32:23.570096 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564156    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 17:32:23.570096 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564159    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564162    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564164    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564167    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564170    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564173    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564176    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564178    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564181    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564184    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564187    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564190    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564192    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564195    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564197    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564200    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564220    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564224    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564226    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 17:32:23.570799 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564229    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564232    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564234    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564238    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564241    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564244    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564246    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564249    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564251    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564254    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564256    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564259    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564261    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564264    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564267    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564269    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564272    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564274    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564277    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 17:32:23.571318 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564279    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564281    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564284    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564287    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564290    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564292    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564295    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564297    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564300    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564302    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564306    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564308    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564311    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564314    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564316    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564319    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564321    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564324    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564326    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564329    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 17:32:23.571915 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564331    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564334    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564336    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564339    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564341    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564343    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564346    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564350    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564353    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564356    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564359    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564362    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564364    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564367    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564370    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564372    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564375    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564378    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564380    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564383    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 17:32:23.572432 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564385    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 17:32:23.573139 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564388    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 17:32:23.573139 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564390    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 17:32:23.573139 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564393    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 17:32:23.573139 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564395    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 17:32:23.573139 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564398    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 17:32:23.573139 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564400    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 17:32:23.573139 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:23.564403    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 17:32:23.573139 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.564408    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 17:32:23.573139 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.564510    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 14 17:32:23.573139 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.568066    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 14 17:32:23.573139 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.569537    2578 server.go:1019] "Starting client certificate rotation"
Jul 14 17:32:23.573139 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.569645    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 17:32:23.573139 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.569697    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 17:32:23.573615 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.571849    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 17:32:23.573868 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.573853    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 17:32:23.581352 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.581334    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 14 17:32:23.585517 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.585499    2578 log.go:25] "Validated CRI v1 image API"
Jul 14 17:32:23.586348 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.586331    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 14 17:32:23.588972 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.588950    2578 fs.go:135] Filesystem UUIDs: map[79942712-fadf-4e3f-98d4-343f508751f9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b145a401-dbf9-499c-a950-2c30159567f5:/dev/nvme0n1p4]
Jul 14 17:32:23.589051 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.588971    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 14 17:32:23.594905 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.594782    2578 manager.go:216] Machine: {Timestamp:2026-07-14 17:32:23.592663346 +0000 UTC m=+0.309057030 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105265 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec280e12fc11309197a1d5da5979f431 SystemUUID:ec280e12-fc11-3091-97a1-d5da5979f431 BootID:7c9fad9d-b633-44dc-b525-816b211dc5d1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:27:fa:3e:0e:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:27:fa:3e:0e:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:89:68:77:be:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 14 17:32:23.594905 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.594897    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 14 17:32:23.595072 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.595014    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 14 17:32:23.597184 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.597166    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 17:32:23.598025 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.598000    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 14 17:32:23.598167 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.598030    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-231.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 14 17:32:23.598806 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.598797    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 14 17:32:23.598838 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.598809    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 14 17:32:23.598838 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.598816    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 17:32:23.599464 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.599454    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 17:32:23.600698 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.600688    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 14 17:32:23.604437 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.604424    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 17:32:23.604576 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.604566    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 14 17:32:23.604606 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.604599    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 14 17:32:23.604652 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.604641    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 14 17:32:23.604683 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.604665    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 14 17:32:23.604683 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.604680    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 14 17:32:23.604762 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.604692    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 14 17:32:23.606387 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.606369    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 17:32:23.606466 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.606396    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 17:32:23.609540 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.609522    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 14 17:32:23.611338 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.611323    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 14 17:32:23.611424 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.611357    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 14 17:32:23.612148 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.612135    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 14 17:32:23.612217 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.612154    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 14 17:32:23.612217 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.612164    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 14 17:32:23.612217 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.612172    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 14 17:32:23.612217 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.612180    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 14 17:32:23.612217 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.612186    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 14 17:32:23.612217 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.612193    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 14 17:32:23.612217 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.612198    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 14 17:32:23.612217 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.612222    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 14 17:32:23.612428 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.612227    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 14 17:32:23.612428 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.612236    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 14 17:32:23.612428 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.612241    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 14 17:32:23.612949 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.612939    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 14 17:32:23.612949 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.612948    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 14 17:32:23.616295 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.616269    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 14 17:32:23.616395 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.616289    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-231.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 14 17:32:23.616395 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.616279    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-231.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 14 17:32:23.616484 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.616472    2578 server.go:1268] "Started kubelet"
Jul 14 17:32:23.616562 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.616538    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 14 17:32:23.616668 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.616613    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 14 17:32:23.616727 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.616686    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 14 17:32:23.617918 ip-10-0-135-231 systemd[1]: Started Kubernetes Kubelet.
Jul 14 17:32:23.618114 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.618098    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 14 17:32:23.619260 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.619244    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 14 17:32:23.622155 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.622124    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 14 17:32:23.623689 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.622782    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-231.ec2.internal.18c23855d0abf9d1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-231.ec2.internal,UID:ip-10-0-135-231.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-231.ec2.internal,},FirstTimestamp:2026-07-14 17:32:23.616444881 +0000 UTC m=+0.332838564,LastTimestamp:2026-07-14 17:32:23.616444881 +0000 UTC m=+0.332838564,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-231.ec2.internal,}"
Jul 14 17:32:23.624113 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.624095    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 14 17:32:23.624226 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.624115    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 14 17:32:23.624226 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.624171    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jul 14 17:32:23.624755 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.624740    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 14 17:32:23.624847 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.624796    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 14 17:32:23.624946 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.624928    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 14 17:32:23.625059 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.625026    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 14 17:32:23.625059 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.625036    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 14 17:32:23.625371 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.625352    2578 factory.go:55] Registering systemd factory
Jul 14 17:32:23.625464 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.625374    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 14 17:32:23.625581 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.625564    2578 factory.go:153] Registering CRI-O factory
Jul 14 17:32:23.625581 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.625583    2578 factory.go:223] Registration of the crio container factory successfully
Jul 14 17:32:23.625709 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.625632    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 14 17:32:23.625709 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.625654    2578 factory.go:103] Registering Raw factory
Jul 14 17:32:23.625709 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.625668    2578 manager.go:1196] Started watching for new ooms in manager
Jul 14 17:32:23.626005 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.625992    2578 manager.go:349] Starting recovery of all containers
Jul 14 17:32:23.627282 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.627255    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-231.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 14 17:32:23.627436 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.627413    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 14 17:32:23.634730 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.634536    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bkb6k"
Jul 14 17:32:23.638016 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.637991    2578 manager.go:354] Recovery completed
Jul 14 17:32:23.639634 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.639611    2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 14 17:32:23.643239 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.643180    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bkb6k"
Jul 14 17:32:23.643574 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.643558    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 17:32:23.645482 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.645465    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 17:32:23.645542 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.645497    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 17:32:23.645542 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.645511    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientPID"
Jul 14 17:32:23.646039 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.646022    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 14 17:32:23.646039 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.646037    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 14 17:32:23.646235 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.646039    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 17:32:23.646235 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.646056    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 17:32:23.646722 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.646707    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 17:32:23.646763 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.646734    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 17:32:23.646763 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.646749    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientPID"
Jul 14 17:32:23.647163 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.647150    2578 policy_none.go:49] "None policy: Start"
Jul 14 17:32:23.647246 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.647169    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 14 17:32:23.647246 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.647183    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 14 17:32:23.647679 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.647593    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-231.ec2.internal.18c23855d2670672  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-231.ec2.internal,UID:ip-10-0-135-231.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-231.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-231.ec2.internal,},FirstTimestamp:2026-07-14 17:32:23.645480562 +0000 UTC m=+0.361874248,LastTimestamp:2026-07-14 17:32:23.645480562 +0000 UTC m=+0.361874248,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-231.ec2.internal,}"
Jul 14 17:32:23.648335 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.648322    2578 policy_none.go:47] "Start"
Jul 14 17:32:23.690857 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.690837    2578 manager.go:337] "Starting Device Plugin manager"
Jul 14 17:32:23.691007 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.690883    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 14 17:32:23.691007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.690897    2578 server.go:87] "Starting device plugin registration server"
Jul 14 17:32:23.691193 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.691177    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 14 17:32:23.691268 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.691193    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 14 17:32:23.691345 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.691328    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 14 17:32:23.691428 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.691418    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 14 17:32:23.691428 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.691429    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 14 17:32:23.692013 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.691995    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 14 17:32:23.692112 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.692048    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-231.ec2.internal\" not found"
Jul 14 17:32:23.774861 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.774829    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 14 17:32:23.776254 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.776238    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 14 17:32:23.776328 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.776281    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 14 17:32:23.776328 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.776313    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 14 17:32:23.776424 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.776356    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 14 17:32:23.778705 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.778682    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 17:32:23.791657 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.791635    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 17:32:23.792512 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.792496    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 17:32:23.792628 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.792532    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 17:32:23.792628 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.792547    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientPID"
Jul 14 17:32:23.792628 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.792577    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-231.ec2.internal"
Jul 14 17:32:23.801395 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.801326    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-231.ec2.internal"
Jul 14 17:32:23.801395 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.801352    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-231.ec2.internal\": node \"ip-10-0-135-231.ec2.internal\" not found"
Jul 14 17:32:23.822391 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.822368    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jul 14 17:32:23.877102 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.877064    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal"]
Jul 14 17:32:23.877281 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.877153    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 17:32:23.878144 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.878129    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 17:32:23.878227 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.878156    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 17:32:23.878227 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.878166    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientPID"
Jul 14 17:32:23.879506 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.879494    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 17:32:23.879619 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.879605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jul 14 17:32:23.879681 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.879634    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 17:32:23.880231 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.880194    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 17:32:23.880342 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.880195    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 17:32:23.880342 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.880267    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 17:32:23.880342 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.880282    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientPID"
Jul 14 17:32:23.880342 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.880240    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 17:32:23.880342 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.880312    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientPID"
Jul 14 17:32:23.881470 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.881454    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal"
Jul 14 17:32:23.881557 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.881483    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 17:32:23.882115 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.882096    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 17:32:23.882185 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.882122    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 17:32:23.882185 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.882131    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientPID"
Jul 14 17:32:23.905864 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.905842    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-231.ec2.internal\" not found" node="ip-10-0-135-231.ec2.internal"
Jul 14 17:32:23.910218 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.910190    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-231.ec2.internal\" not found" node="ip-10-0-135-231.ec2.internal"
Jul 14 17:32:23.922644 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:23.922628    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jul 14 17:32:23.927012 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.926996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99dd3a4d2ab0d3d55d4198e9e5c8c72d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal\" (UID: \"99dd3a4d2ab0d3d55d4198e9e5c8c72d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jul 14 17:32:23.927083 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.927019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8a89385d90f63ff945cda7868eed5df-config\") pod \"kube-apiserver-proxy-ip-10-0-135-231.ec2.internal\" (UID: \"e8a89385d90f63ff945cda7868eed5df\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal"
Jul 14 17:32:23.927083 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:23.927038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99dd3a4d2ab0d3d55d4198e9e5c8c72d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal\" (UID: \"99dd3a4d2ab0d3d55d4198e9e5c8c72d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jul 14 17:32:24.023438 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.023410    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jul 14 17:32:24.027738 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.027720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8a89385d90f63ff945cda7868eed5df-config\") pod \"kube-apiserver-proxy-ip-10-0-135-231.ec2.internal\" (UID: \"e8a89385d90f63ff945cda7868eed5df\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal"
Jul 14 17:32:24.027791 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.027754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99dd3a4d2ab0d3d55d4198e9e5c8c72d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal\" (UID: \"99dd3a4d2ab0d3d55d4198e9e5c8c72d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jul 14 17:32:24.027791 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.027779    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8a89385d90f63ff945cda7868eed5df-config\") pod \"kube-apiserver-proxy-ip-10-0-135-231.ec2.internal\" (UID: \"e8a89385d90f63ff945cda7868eed5df\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal"
Jul 14 17:32:24.027869 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.027804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99dd3a4d2ab0d3d55d4198e9e5c8c72d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal\" (UID: \"99dd3a4d2ab0d3d55d4198e9e5c8c72d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jul 14 17:32:24.027869 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.027809    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99dd3a4d2ab0d3d55d4198e9e5c8c72d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal\" (UID: \"99dd3a4d2ab0d3d55d4198e9e5c8c72d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jul 14 17:32:24.027869 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.027778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99dd3a4d2ab0d3d55d4198e9e5c8c72d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal\" (UID: \"99dd3a4d2ab0d3d55d4198e9e5c8c72d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jul 14 17:32:24.124141 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.124078    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jul 14 17:32:24.209509 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.209480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jul 14 17:32:24.214849 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.214825    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal"
Jul 14 17:32:24.224368 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.224344    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jul 14 17:32:24.324811 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.324765    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jul 14 17:32:24.425670 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.425625    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jul 14 17:32:24.449271 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.449251    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 17:32:24.529473 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.529442    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jul 14 17:32:24.570823 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.570799    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 14 17:32:24.571478 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.570941    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 17:32:24.571478 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.570941    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 17:32:24.583148 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.583124    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 17:32:24.605536 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.605508    2578 apiserver.go:52] "Watching apiserver"
Jul 14 17:32:24.615631 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.615610    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 14 17:32:24.615959 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.615937    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-fdndd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2","openshift-cluster-node-tuning-operator/tuned-rcnm5","openshift-dns/node-resolver-786wn","openshift-multus/network-metrics-daemon-x8dr4","openshift-network-diagnostics/network-check-target-hv7kv","kube-system/global-pull-secret-syncer-fk7h5","openshift-image-registry/node-ca-554b4","openshift-multus/multus-additional-cni-plugins-6wfqp","openshift-multus/multus-lftjp","openshift-network-operator/iptables-alerter-6xckn","openshift-ovn-kubernetes/ovnkube-node-g9hr5"]
Jul 14 17:32:24.617581 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.617565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fdndd"
Jul 14 17:32:24.618786 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.618770    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.620070 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.620051    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.620502 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.620484    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-49xgx\""
Jul 14 17:32:24.620571 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.620514    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 14 17:32:24.621154 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.621131    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 14 17:32:24.621269 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.621217    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 14 17:32:24.621451 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.621432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-786wn"
Jul 14 17:32:24.621514 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.621482    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 14 17:32:24.621704 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.621688    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bvqph\""
Jul 14 17:32:24.622231 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.622217    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 14 17:32:24.622564 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.622546    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 14 17:32:24.622667 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.622631    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:24.622667 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.622634    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 14 17:32:24.622777 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.622692    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:24.623795 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.623778    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-77cpz\""
Jul 14 17:32:24.623888 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.623871    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:24.623950 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.623937    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:24.624663 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.624646    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 14 17:32:24.624757 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.624651    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jul 14 17:32:24.624899 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.624877    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 14 17:32:24.625025 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.624983    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 14 17:32:24.625165 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.625153    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wkdqr\""
Jul 14 17:32:24.625388 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.625373    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:24.625452 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.625427    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:24.625529 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.625507    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 17:32:24.626721 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.626706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-554b4"
Jul 14 17:32:24.628671 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.628122    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.630459 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.630041    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 14 17:32:24.630459 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.630181    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 14 17:32:24.630459 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.630380    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.630658 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.630645    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8fsq9\""
Jul 14 17:32:24.630833 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.630749    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-host\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f715504c-a483-4450-838c-06e8d0c57b4c-etc-tuned\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f715504c-a483-4450-838c-06e8d0c57b4c-tmp\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs\") pod \"network-metrics-daemon-x8dr4\" (UID: \"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d\") " pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-kubelet-config\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86jn2\" (UniqueName: \"kubernetes.io/projected/f715504c-a483-4450-838c-06e8d0c57b4c-kube-api-access-86jn2\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52e52a4c-3f78-40f6-8d9e-963ff65c0f86-hosts-file\") pod \"node-resolver-786wn\" (UID: \"52e52a4c-3f78-40f6-8d9e-963ff65c0f86\") " pod="openshift-dns/node-resolver-786wn"
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631551    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-socket-dir\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-device-dir\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631578    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-systemd\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631681    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-lib-modules\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e52a4c-3f78-40f6-8d9e-963ff65c0f86-tmp-dir\") pod \"node-resolver-786wn\" (UID: \"52e52a4c-3f78-40f6-8d9e-963ff65c0f86\") " pod="openshift-dns/node-resolver-786wn"
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8qtq\" (UniqueName: \"kubernetes.io/projected/52e52a4c-3f78-40f6-8d9e-963ff65c0f86-kube-api-access-w8qtq\") pod \"node-resolver-786wn\" (UID: \"52e52a4c-3f78-40f6-8d9e-963ff65c0f86\") " pod="openshift-dns/node-resolver-786wn"
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-dbus\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631761    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 14 17:32:24.631778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-modprobe-d\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631820    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r8q82\""
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-sys-fs\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-run\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-sys\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.631976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4xpb\" (UniqueName: \"kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb\") pod \"network-check-target-hv7kv\" (UID: \"ab91d593-f6ad-4a22-9d7f-e90a0dcc9787\") " pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.632002    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.632024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-sysctl-conf\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.632030    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.632045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5afed38b-3690-4505-9749-3d026e206a4f-konnectivity-ca\") pod \"konnectivity-agent-fdndd\" (UID: \"5afed38b-3690-4505-9749-3d026e206a4f\") " pod="kube-system/konnectivity-agent-fdndd"
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.632085    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-registration-dir\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.632125    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.632126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdxht\" (UniqueName: \"kubernetes.io/projected/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-kube-api-access-bdxht\") pod \"network-metrics-daemon-x8dr4\" (UID: \"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d\") " pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.632172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-sysctl-d\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.632197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5afed38b-3690-4505-9749-3d026e206a4f-agent-certs\") pod \"konnectivity-agent-fdndd\" (UID: \"5afed38b-3690-4505-9749-3d026e206a4f\") " pod="kube-system/konnectivity-agent-fdndd"
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.632243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74kzt\" (UniqueName: \"kubernetes.io/projected/4349f767-b8be-4113-b166-607fdcd075ca-kube-api-access-74kzt\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.632283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.632328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-sysconfig\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.632901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.632358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-kubernetes\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.633772 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.632381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-var-lib-kubelet\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.633772 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.632926    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6xckn"
Jul 14 17:32:24.633772 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.633416    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 14 17:32:24.634003 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.633981    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sqmxw\""
Jul 14 17:32:24.634202 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.634186    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 14 17:32:24.634606 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.634586    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.634976 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.634954    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 14 17:32:24.635786 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.635765    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 14 17:32:24.635870 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.635831    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 14 17:32:24.635945 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.635887    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mth2q\""
Jul 14 17:32:24.637395 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.637379    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 14 17:32:24.637494 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.637379    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 14 17:32:24.637494 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.637452    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g7dzq\""
Jul 14 17:32:24.637817 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.637801    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 14 17:32:24.637891 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.637847    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 14 17:32:24.637891 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.637810    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 14 17:32:24.637994 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.637912    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 14 17:32:24.640481 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.640460    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"]
Jul 14 17:32:24.640633 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.640620    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 17:32:24.640737 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.640724    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal"
Jul 14 17:32:24.642613 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.642598    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 17:32:24.647946 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.647900    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 17:27:23 +0000 UTC" deadline="2028-04-05 05:28:45.097032792 +0000 UTC"
Jul 14 17:32:24.647946 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.647934    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15131h56m20.449100719s"
Jul 14 17:32:24.649013 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.648993    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal"]
Jul 14 17:32:24.649362 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.649347    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 17:32:24.664913 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.664864    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-htkgv"
Jul 14 17:32:24.674483 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.674468    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-htkgv"
Jul 14 17:32:24.726223 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.726198    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 14 17:32:24.733273 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-systemd\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.733273 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-lib-modules\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.733459 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e52a4c-3f78-40f6-8d9e-963ff65c0f86-tmp-dir\") pod \"node-resolver-786wn\" (UID: \"52e52a4c-3f78-40f6-8d9e-963ff65c0f86\") " pod="openshift-dns/node-resolver-786wn"
Jul 14 17:32:24.733459 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qtq\" (UniqueName: \"kubernetes.io/projected/52e52a4c-3f78-40f6-8d9e-963ff65c0f86-kube-api-access-w8qtq\") pod \"node-resolver-786wn\" (UID: \"52e52a4c-3f78-40f6-8d9e-963ff65c0f86\") " pod="openshift-dns/node-resolver-786wn"
Jul 14 17:32:24.733459 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733371    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-systemd\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.733459 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-dbus\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:24.733459 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-multus-daemon-config\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.733662 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-lib-modules\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.733662 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-dbus\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:24.733662 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e16a2359-cd0f-4b37-99d9-938bd35c13f7-ovnkube-config\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.733662 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-os-release\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.733662 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-var-lib-cni-bin\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.733662 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-etc-kubernetes\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.733662 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-run\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.733662 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e52a4c-3f78-40f6-8d9e-963ff65c0f86-tmp-dir\") pod \"node-resolver-786wn\" (UID: \"52e52a4c-3f78-40f6-8d9e-963ff65c0f86\") " pod="openshift-dns/node-resolver-786wn"
Jul 14 17:32:24.733662 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-sys\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.733662 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xpb\" (UniqueName: \"kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb\") pod \"network-check-target-hv7kv\" (UID: \"ab91d593-f6ad-4a22-9d7f-e90a0dcc9787\") " pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:24.734100 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab594791-63ca-446f-b89b-7e8c7724b7e7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.734100 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733698    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-run\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.734100 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-sys\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.734100 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-cnibin\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.734100 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-run-k8s-cni-cncf-io\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.734100 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.734408 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.733836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab594791-63ca-446f-b89b-7e8c7724b7e7-os-release\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.734408 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.734277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ab594791-63ca-446f-b89b-7e8c7724b7e7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.734408 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.734313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7792d2f3-ffbe-4009-8459-c38262ce3c30-iptables-alerter-script\") pod \"iptables-alerter-6xckn\" (UID: \"7792d2f3-ffbe-4009-8459-c38262ce3c30\") " pod="openshift-network-operator/iptables-alerter-6xckn"
Jul 14 17:32:24.734408 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.734363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f85b40b-bca9-4452-ac81-5786ca39e3e4-host\") pod \"node-ca-554b4\" (UID: \"6f85b40b-bca9-4452-ac81-5786ca39e3e4\") " pod="openshift-image-registry/node-ca-554b4"
Jul 14 17:32:24.734599 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.734424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f85b40b-bca9-4452-ac81-5786ca39e3e4-serviceca\") pod \"node-ca-554b4\" (UID: \"6f85b40b-bca9-4452-ac81-5786ca39e3e4\") " pod="openshift-image-registry/node-ca-554b4"
Jul 14 17:32:24.734599 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.734547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-sysctl-d\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.734599 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.734585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74kzt\" (UniqueName: \"kubernetes.io/projected/4349f767-b8be-4113-b166-607fdcd075ca-kube-api-access-74kzt\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.734801 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.734623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:24.734801 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.734664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e16a2359-cd0f-4b37-99d9-938bd35c13f7-ovn-node-metrics-cert\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.734801 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.734781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-sysctl-d\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.735007    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e16a2359-cd0f-4b37-99d9-938bd35c13f7-ovnkube-script-lib\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.735236    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret podName:fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7 nodeName:}" failed. No retries permitted until 2026-07-14 17:32:25.235204747 +0000 UTC m=+1.951598420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret") pod "global-pull-secret-syncer-fk7h5" (UID: "fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7") : object "kube-system"/"original-pull-secret" not registered
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab594791-63ca-446f-b89b-7e8c7724b7e7-cnibin\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-var-lib-cni-multus\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-slash\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-kubernetes\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-host\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs\") pod \"network-metrics-daemon-x8dr4\" (UID: \"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d\") " pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-run-netns\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-etc-openvswitch\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-run-openvswitch\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-kubelet-config\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-multus-cni-dir\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-multus-conf-dir\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.735668    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:32:24.737319 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52e52a4c-3f78-40f6-8d9e-963ff65c0f86-hosts-file\") pod \"node-resolver-786wn\" (UID: \"52e52a4c-3f78-40f6-8d9e-963ff65c0f86\") " pod="openshift-dns/node-resolver-786wn"
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.735734    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs podName:535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d nodeName:}" failed. No retries permitted until 2026-07-14 17:32:25.235705854 +0000 UTC m=+1.952099541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs") pod "network-metrics-daemon-x8dr4" (UID: "535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735815    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-kubelet-config\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-kubernetes\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52e52a4c-3f78-40f6-8d9e-963ff65c0f86-hosts-file\") pod \"node-resolver-786wn\" (UID: \"52e52a4c-3f78-40f6-8d9e-963ff65c0f86\") " pod="openshift-dns/node-resolver-786wn"
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.735943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-socket-dir\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-host\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-device-dir\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-run-systemd\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv4md\" (UniqueName: \"kubernetes.io/projected/7792d2f3-ffbe-4009-8459-c38262ce3c30-kube-api-access-gv4md\") pod \"iptables-alerter-6xckn\" (UID: \"7792d2f3-ffbe-4009-8459-c38262ce3c30\") " pod="openshift-network-operator/iptables-alerter-6xckn"
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-system-cni-dir\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-run-multus-certs\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-systemd-units\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-modprobe-d\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-sys-fs\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.738203 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-run-netns\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-log-socket\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-cni-bin\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ab594791-63ca-446f-b89b-7e8c7724b7e7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrbv6\" (UniqueName: \"kubernetes.io/projected/6f85b40b-bca9-4452-ac81-5786ca39e3e4-kube-api-access-mrbv6\") pod \"node-ca-554b4\" (UID: \"6f85b40b-bca9-4452-ac81-5786ca39e3e4\") " pod="openshift-image-registry/node-ca-554b4"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-var-lib-openvswitch\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-node-log\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7792d2f3-ffbe-4009-8459-c38262ce3c30-host-slash\") pod \"iptables-alerter-6xckn\" (UID: \"7792d2f3-ffbe-4009-8459-c38262ce3c30\") " pod="openshift-network-operator/iptables-alerter-6xckn"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.736951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-modprobe-d\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.737012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-sysctl-conf\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.737047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5afed38b-3690-4505-9749-3d026e206a4f-konnectivity-ca\") pod \"konnectivity-agent-fdndd\" (UID: \"5afed38b-3690-4505-9749-3d026e206a4f\") " pod="kube-system/konnectivity-agent-fdndd"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.737052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.737102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-registration-dir\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.737140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdxht\" (UniqueName: \"kubernetes.io/projected/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-kube-api-access-bdxht\") pod \"network-metrics-daemon-x8dr4\" (UID: \"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d\") " pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:24.739094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.737240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-cni-binary-copy\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.737271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab594791-63ca-446f-b89b-7e8c7724b7e7-system-cni-dir\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.737962    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5afed38b-3690-4505-9749-3d026e206a4f-konnectivity-ca\") pod \"konnectivity-agent-fdndd\" (UID: \"5afed38b-3690-4505-9749-3d026e206a4f\") " pod="kube-system/konnectivity-agent-fdndd"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.738234    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-device-dir\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.738248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.738304    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-sys-fs\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.738297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-socket-dir\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.738416    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-sysctl-conf\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.738471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5afed38b-3690-4505-9749-3d026e206a4f-agent-certs\") pod \"konnectivity-agent-fdndd\" (UID: \"5afed38b-3690-4505-9749-3d026e206a4f\") " pod="kube-system/konnectivity-agent-fdndd"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.738475    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4349f767-b8be-4113-b166-607fdcd075ca-registration-dir\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.738552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e16a2359-cd0f-4b37-99d9-938bd35c13f7-env-overrides\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.738696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-var-lib-kubelet\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.738739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-sysconfig\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-var-lib-kubelet\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739149    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-etc-sysconfig\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739177    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f715504c-a483-4450-838c-06e8d0c57b4c-var-lib-kubelet\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.739910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f715504c-a483-4450-838c-06e8d0c57b4c-etc-tuned\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.740937 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f715504c-a483-4450-838c-06e8d0c57b4c-tmp\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.740937 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-multus-socket-dir-parent\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.740937 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-hostroot\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.740937 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-kubelet\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.740937 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-run-ovn\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.740937 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6fjm\" (UniqueName: \"kubernetes.io/projected/ab594791-63ca-446f-b89b-7e8c7724b7e7-kube-api-access-p6fjm\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.740937 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86jn2\" (UniqueName: \"kubernetes.io/projected/f715504c-a483-4450-838c-06e8d0c57b4c-kube-api-access-86jn2\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.740937 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62mnm\" (UniqueName: \"kubernetes.io/projected/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-kube-api-access-62mnm\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.740937 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-cni-netd\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.740937 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssgsh\" (UniqueName: \"kubernetes.io/projected/e16a2359-cd0f-4b37-99d9-938bd35c13f7-kube-api-access-ssgsh\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.740937 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.739618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab594791-63ca-446f-b89b-7e8c7724b7e7-cni-binary-copy\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.740937 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.740721    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 17:32:24.740937 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.740748    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 17:32:24.740937 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.740762    2578 projected.go:196] Error preparing data for projected volume kube-api-access-b4xpb for pod openshift-network-diagnostics/network-check-target-hv7kv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:32:24.740937 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:24.740826    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb podName:ab91d593-f6ad-4a22-9d7f-e90a0dcc9787 nodeName:}" failed. No retries permitted until 2026-07-14 17:32:25.24080866 +0000 UTC m=+1.957202353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b4xpb" (UniqueName: "kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb") pod "network-check-target-hv7kv" (UID: "ab91d593-f6ad-4a22-9d7f-e90a0dcc9787") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:32:24.742481 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.742458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f715504c-a483-4450-838c-06e8d0c57b4c-etc-tuned\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.742481 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.742478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f715504c-a483-4450-838c-06e8d0c57b4c-tmp\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.742656 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.742639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5afed38b-3690-4505-9749-3d026e206a4f-agent-certs\") pod \"konnectivity-agent-fdndd\" (UID: \"5afed38b-3690-4505-9749-3d026e206a4f\") " pod="kube-system/konnectivity-agent-fdndd"
Jul 14 17:32:24.743589 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.743571    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8qtq\" (UniqueName: \"kubernetes.io/projected/52e52a4c-3f78-40f6-8d9e-963ff65c0f86-kube-api-access-w8qtq\") pod \"node-resolver-786wn\" (UID: \"52e52a4c-3f78-40f6-8d9e-963ff65c0f86\") " pod="openshift-dns/node-resolver-786wn"
Jul 14 17:32:24.747141 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.747062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74kzt\" (UniqueName: \"kubernetes.io/projected/4349f767-b8be-4113-b166-607fdcd075ca-kube-api-access-74kzt\") pod \"aws-ebs-csi-driver-node-492s2\" (UID: \"4349f767-b8be-4113-b166-607fdcd075ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.747696 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.747677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdxht\" (UniqueName: \"kubernetes.io/projected/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-kube-api-access-bdxht\") pod \"network-metrics-daemon-x8dr4\" (UID: \"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d\") " pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:24.748625 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.748606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86jn2\" (UniqueName: \"kubernetes.io/projected/f715504c-a483-4450-838c-06e8d0c57b4c-kube-api-access-86jn2\") pod \"tuned-rcnm5\" (UID: \"f715504c-a483-4450-838c-06e8d0c57b4c\") " pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.839929 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.839900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62mnm\" (UniqueName: \"kubernetes.io/projected/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-kube-api-access-62mnm\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.840112 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.839934    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-cni-netd\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.840112 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.839957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssgsh\" (UniqueName: \"kubernetes.io/projected/e16a2359-cd0f-4b37-99d9-938bd35c13f7-kube-api-access-ssgsh\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.840112 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.839983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab594791-63ca-446f-b89b-7e8c7724b7e7-cni-binary-copy\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.840112 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-cni-netd\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.840358 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-multus-daemon-config\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.840358 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e16a2359-cd0f-4b37-99d9-938bd35c13f7-ovnkube-config\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.840358 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-os-release\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.840358 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-var-lib-cni-bin\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.840358 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-etc-kubernetes\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.840541 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-var-lib-cni-bin\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.840541 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab594791-63ca-446f-b89b-7e8c7724b7e7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.840541 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-cnibin\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.840541 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-run-k8s-cni-cncf-io\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.840541 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.840541 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab594791-63ca-446f-b89b-7e8c7724b7e7-os-release\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.840541 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ab594791-63ca-446f-b89b-7e8c7724b7e7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.840812 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7792d2f3-ffbe-4009-8459-c38262ce3c30-iptables-alerter-script\") pod \"iptables-alerter-6xckn\" (UID: \"7792d2f3-ffbe-4009-8459-c38262ce3c30\") " pod="openshift-network-operator/iptables-alerter-6xckn"
Jul 14 17:32:24.840812 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab594791-63ca-446f-b89b-7e8c7724b7e7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.840812 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f85b40b-bca9-4452-ac81-5786ca39e3e4-host\") pod \"node-ca-554b4\" (UID: \"6f85b40b-bca9-4452-ac81-5786ca39e3e4\") " pod="openshift-image-registry/node-ca-554b4"
Jul 14 17:32:24.840812 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-etc-kubernetes\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.840812 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840610    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f85b40b-bca9-4452-ac81-5786ca39e3e4-serviceca\") pod \"node-ca-554b4\" (UID: \"6f85b40b-bca9-4452-ac81-5786ca39e3e4\") " pod="openshift-image-registry/node-ca-554b4"
Jul 14 17:32:24.840812 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e16a2359-cd0f-4b37-99d9-938bd35c13f7-ovn-node-metrics-cert\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.840812 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e16a2359-cd0f-4b37-99d9-938bd35c13f7-ovnkube-script-lib\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.840812 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840698    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab594791-63ca-446f-b89b-7e8c7724b7e7-cni-binary-copy\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.840812 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab594791-63ca-446f-b89b-7e8c7724b7e7-cnibin\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.840812 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-var-lib-cni-multus\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.840812 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-slash\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.840812 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-run-netns\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.840812 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-etc-openvswitch\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-run-netns\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840880    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-var-lib-cni-multus\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-slash\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-run-openvswitch\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840376    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-os-release\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-etc-openvswitch\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-multus-cni-dir\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e16a2359-cd0f-4b37-99d9-938bd35c13f7-ovnkube-config\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab594791-63ca-446f-b89b-7e8c7724b7e7-cnibin\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.840989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-cnibin\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-multus-conf-dir\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-run-k8s-cni-cncf-io\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-run-systemd\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f85b40b-bca9-4452-ac81-5786ca39e3e4-serviceca\") pod \"node-ca-554b4\" (UID: \"6f85b40b-bca9-4452-ac81-5786ca39e3e4\") " pod="openshift-image-registry/node-ca-554b4"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv4md\" (UniqueName: \"kubernetes.io/projected/7792d2f3-ffbe-4009-8459-c38262ce3c30-kube-api-access-gv4md\") pod \"iptables-alerter-6xckn\" (UID: \"7792d2f3-ffbe-4009-8459-c38262ce3c30\") " pod="openshift-network-operator/iptables-alerter-6xckn"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-run-openvswitch\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.841366 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-run-systemd\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-multus-conf-dir\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-system-cni-dir\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841116    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab594791-63ca-446f-b89b-7e8c7724b7e7-os-release\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-run-multus-certs\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-systemd-units\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-run-netns\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-log-socket\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-cni-bin\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ab594791-63ca-446f-b89b-7e8c7724b7e7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrbv6\" (UniqueName: \"kubernetes.io/projected/6f85b40b-bca9-4452-ac81-5786ca39e3e4-kube-api-access-mrbv6\") pod \"node-ca-554b4\" (UID: \"6f85b40b-bca9-4452-ac81-5786ca39e3e4\") " pod="openshift-image-registry/node-ca-554b4"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-var-lib-openvswitch\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-node-log\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7792d2f3-ffbe-4009-8459-c38262ce3c30-host-slash\") pod \"iptables-alerter-6xckn\" (UID: \"7792d2f3-ffbe-4009-8459-c38262ce3c30\") " pod="openshift-network-operator/iptables-alerter-6xckn"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-cni-binary-copy\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab594791-63ca-446f-b89b-7e8c7724b7e7-system-cni-dir\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.842159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841517    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e16a2359-cd0f-4b37-99d9-938bd35c13f7-ovnkube-script-lib\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e16a2359-cd0f-4b37-99d9-938bd35c13f7-env-overrides\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-var-lib-kubelet\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-multus-socket-dir-parent\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ab594791-63ca-446f-b89b-7e8c7724b7e7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-hostroot\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-kubelet\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841680    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-run-ovn\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841707    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fjm\" (UniqueName: \"kubernetes.io/projected/ab594791-63ca-446f-b89b-7e8c7724b7e7-kube-api-access-p6fjm\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-multus-cni-dir\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.841995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-var-lib-kubelet\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842044    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-multus-socket-dir-parent\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842083    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-hostroot\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-kubelet\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-run-ovn\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842170    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7792d2f3-ffbe-4009-8459-c38262ce3c30-iptables-alerter-script\") pod \"iptables-alerter-6xckn\" (UID: \"7792d2f3-ffbe-4009-8459-c38262ce3c30\") " pod="openshift-network-operator/iptables-alerter-6xckn"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f85b40b-bca9-4452-ac81-5786ca39e3e4-host\") pod \"node-ca-554b4\" (UID: \"6f85b40b-bca9-4452-ac81-5786ca39e3e4\") " pod="openshift-image-registry/node-ca-554b4"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842269    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-multus-daemon-config\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.843040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-systemd-units\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.843805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842335    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-log-socket\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.843805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-system-cni-dir\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.843805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-run-netns\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.843805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842395    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-host-run-multus-certs\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.843805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842421    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.843805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-var-lib-openvswitch\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.843805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-node-log\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.843805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ab594791-63ca-446f-b89b-7e8c7724b7e7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.843805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab594791-63ca-446f-b89b-7e8c7724b7e7-system-cni-dir\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.843805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7792d2f3-ffbe-4009-8459-c38262ce3c30-host-slash\") pod \"iptables-alerter-6xckn\" (UID: \"7792d2f3-ffbe-4009-8459-c38262ce3c30\") " pod="openshift-network-operator/iptables-alerter-6xckn"
Jul 14 17:32:24.843805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842590    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e16a2359-cd0f-4b37-99d9-938bd35c13f7-env-overrides\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.843805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.842360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e16a2359-cd0f-4b37-99d9-938bd35c13f7-host-cni-bin\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.843805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.843680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e16a2359-cd0f-4b37-99d9-938bd35c13f7-ovn-node-metrics-cert\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.843805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.843690    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-cni-binary-copy\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.851148 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.851119    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62mnm\" (UniqueName: \"kubernetes.io/projected/975b9e78-ce91-4d64-8ac5-0996dbdeab4a-kube-api-access-62mnm\") pod \"multus-lftjp\" (UID: \"975b9e78-ce91-4d64-8ac5-0996dbdeab4a\") " pod="openshift-multus/multus-lftjp"
Jul 14 17:32:24.851471 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.851451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv4md\" (UniqueName: \"kubernetes.io/projected/7792d2f3-ffbe-4009-8459-c38262ce3c30-kube-api-access-gv4md\") pod \"iptables-alerter-6xckn\" (UID: \"7792d2f3-ffbe-4009-8459-c38262ce3c30\") " pod="openshift-network-operator/iptables-alerter-6xckn"
Jul 14 17:32:24.851550 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.851479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssgsh\" (UniqueName: \"kubernetes.io/projected/e16a2359-cd0f-4b37-99d9-938bd35c13f7-kube-api-access-ssgsh\") pod \"ovnkube-node-g9hr5\" (UID: \"e16a2359-cd0f-4b37-99d9-938bd35c13f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:24.851550 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.851502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6fjm\" (UniqueName: \"kubernetes.io/projected/ab594791-63ca-446f-b89b-7e8c7724b7e7-kube-api-access-p6fjm\") pod \"multus-additional-cni-plugins-6wfqp\" (UID: \"ab594791-63ca-446f-b89b-7e8c7724b7e7\") " pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:24.852033 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.852012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrbv6\" (UniqueName: \"kubernetes.io/projected/6f85b40b-bca9-4452-ac81-5786ca39e3e4-kube-api-access-mrbv6\") pod \"node-ca-554b4\" (UID: \"6f85b40b-bca9-4452-ac81-5786ca39e3e4\") " pod="openshift-image-registry/node-ca-554b4"
Jul 14 17:32:24.929474 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:24.929424    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99dd3a4d2ab0d3d55d4198e9e5c8c72d.slice/crio-1509f0197a10e57d6ab30c4ed6483dee9a41cbae92e4d0c00ac4a64d0f10f858 WatchSource:0}: Error finding container 1509f0197a10e57d6ab30c4ed6483dee9a41cbae92e4d0c00ac4a64d0f10f858: Status 404 returned error can't find the container with id 1509f0197a10e57d6ab30c4ed6483dee9a41cbae92e4d0c00ac4a64d0f10f858
Jul 14 17:32:24.932839 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.932822    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 17:32:24.952885 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.952853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fdndd"
Jul 14 17:32:24.962139 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:24.962112    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afed38b_3690_4505_9749_3d026e206a4f.slice/crio-fe0e684323f9564b274d01f288b772bc107b3df9850eba9a67c30333dd377ff1 WatchSource:0}: Error finding container fe0e684323f9564b274d01f288b772bc107b3df9850eba9a67c30333dd377ff1: Status 404 returned error can't find the container with id fe0e684323f9564b274d01f288b772bc107b3df9850eba9a67c30333dd377ff1
Jul 14 17:32:24.967592 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.967572    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2"
Jul 14 17:32:24.975149 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:24.975119    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4349f767_b8be_4113_b166_607fdcd075ca.slice/crio-04c21c676607f476e33560308ad7e1b2bf43182ab679f78690a17c7c4be3ff55 WatchSource:0}: Error finding container 04c21c676607f476e33560308ad7e1b2bf43182ab679f78690a17c7c4be3ff55: Status 404 returned error can't find the container with id 04c21c676607f476e33560308ad7e1b2bf43182ab679f78690a17c7c4be3ff55
Jul 14 17:32:24.989727 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:24.989709    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rcnm5"
Jul 14 17:32:24.997931 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:24.997911    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf715504c_a483_4450_838c_06e8d0c57b4c.slice/crio-071d60a6aa9503a0a160097bd0b70be42461d7efaefc57b343ae19c3681b89e9 WatchSource:0}: Error finding container 071d60a6aa9503a0a160097bd0b70be42461d7efaefc57b343ae19c3681b89e9: Status 404 returned error can't find the container with id 071d60a6aa9503a0a160097bd0b70be42461d7efaefc57b343ae19c3681b89e9
Jul 14 17:32:25.005890 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.005874    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-786wn"
Jul 14 17:32:25.014101 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.014082    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-554b4"
Jul 14 17:32:25.015211 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:25.015186    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e52a4c_3f78_40f6_8d9e_963ff65c0f86.slice/crio-1dee10b3d7c16548f559c653c2a4d3124436eb1ffd6beb012c55118b84f8c3b4 WatchSource:0}: Error finding container 1dee10b3d7c16548f559c653c2a4d3124436eb1ffd6beb012c55118b84f8c3b4: Status 404 returned error can't find the container with id 1dee10b3d7c16548f559c653c2a4d3124436eb1ffd6beb012c55118b84f8c3b4
Jul 14 17:32:25.023189 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:25.023165    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f85b40b_bca9_4452_ac81_5786ca39e3e4.slice/crio-2496334e86a8ab70c54873cc86a1236761eadb4e88bb7d66666c9cac3dcbafbc WatchSource:0}: Error finding container 2496334e86a8ab70c54873cc86a1236761eadb4e88bb7d66666c9cac3dcbafbc: Status 404 returned error can't find the container with id 2496334e86a8ab70c54873cc86a1236761eadb4e88bb7d66666c9cac3dcbafbc
Jul 14 17:32:25.029643 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.029626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6wfqp"
Jul 14 17:32:25.041148 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.041130    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lftjp"
Jul 14 17:32:25.041595 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:25.041568    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab594791_63ca_446f_b89b_7e8c7724b7e7.slice/crio-eb4ab218b491530727f838dc3c23b7f27e096a230b5b6a8efc3c65987121c7df WatchSource:0}: Error finding container eb4ab218b491530727f838dc3c23b7f27e096a230b5b6a8efc3c65987121c7df: Status 404 returned error can't find the container with id eb4ab218b491530727f838dc3c23b7f27e096a230b5b6a8efc3c65987121c7df
Jul 14 17:32:25.042364 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:25.042338    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a89385d90f63ff945cda7868eed5df.slice/crio-d84c7347ce9b15360928de01cbdb9aa5ecb2cecb4e2bbecd4e6cdbf9133417bc WatchSource:0}: Error finding container d84c7347ce9b15360928de01cbdb9aa5ecb2cecb4e2bbecd4e6cdbf9133417bc: Status 404 returned error can't find the container with id d84c7347ce9b15360928de01cbdb9aa5ecb2cecb4e2bbecd4e6cdbf9133417bc
Jul 14 17:32:25.048676 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:25.048657    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975b9e78_ce91_4d64_8ac5_0996dbdeab4a.slice/crio-b7694574c968a7ce39341a6ca4a26a73ecaaeb003ee796490c446291e7e8ee8d WatchSource:0}: Error finding container b7694574c968a7ce39341a6ca4a26a73ecaaeb003ee796490c446291e7e8ee8d: Status 404 returned error can't find the container with id b7694574c968a7ce39341a6ca4a26a73ecaaeb003ee796490c446291e7e8ee8d
Jul 14 17:32:25.065644 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.065626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6xckn"
Jul 14 17:32:25.070284 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.070267    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:25.073460 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:25.073440    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7792d2f3_ffbe_4009_8459_c38262ce3c30.slice/crio-c3272769b1d834d2b04c393f3bbfb799bd9fff63a264faefd17d88cb87fb449b WatchSource:0}: Error finding container c3272769b1d834d2b04c393f3bbfb799bd9fff63a264faefd17d88cb87fb449b: Status 404 returned error can't find the container with id c3272769b1d834d2b04c393f3bbfb799bd9fff63a264faefd17d88cb87fb449b
Jul 14 17:32:25.078332 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:32:25.078310    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16a2359_cd0f_4b37_99d9_938bd35c13f7.slice/crio-26501c8e017afd111d0fef5d308774a4c6902654cbab040ed5767ae1c5b0203e WatchSource:0}: Error finding container 26501c8e017afd111d0fef5d308774a4c6902654cbab040ed5767ae1c5b0203e: Status 404 returned error can't find the container with id 26501c8e017afd111d0fef5d308774a4c6902654cbab040ed5767ae1c5b0203e
Jul 14 17:32:25.080863 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:25.080846    2578 kuberuntime_manager.go:1459] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-g9hr5_openshift-ovn-kubernetes(e16a2359-cd0f-4b37-99d9-938bd35c13f7): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 14 17:32:25.082211 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:25.082194    2578 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-g9hr5_openshift-ovn-kubernetes(e16a2359-cd0f-4b37-99d9-938bd35c13f7): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 14 17:32:25.087965 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:25.087925    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d1573db16143027c8ffaf46a01ce12f2bcd89c15410c42ca54be4b3b1ddfdd6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5" podUID="e16a2359-cd0f-4b37-99d9-938bd35c13f7"
Jul 14 17:32:25.244440 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.244354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xpb\" (UniqueName: \"kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb\") pod \"network-check-target-hv7kv\" (UID: \"ab91d593-f6ad-4a22-9d7f-e90a0dcc9787\") " pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:25.244440 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.244398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:25.244440 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.244421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs\") pod \"network-metrics-daemon-x8dr4\" (UID: \"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d\") " pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:25.244661 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:25.244511    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:32:25.244661 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:25.244516    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 17:32:25.244661 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:25.244519    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 17:32:25.244661 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:25.244582    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 17:32:25.244661 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:25.244593    2578 projected.go:196] Error preparing data for projected volume kube-api-access-b4xpb for pod openshift-network-diagnostics/network-check-target-hv7kv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:32:25.244661 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:25.244565    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs podName:535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d nodeName:}" failed. No retries permitted until 2026-07-14 17:32:26.244549433 +0000 UTC m=+2.960943104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs") pod "network-metrics-daemon-x8dr4" (UID: "535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:32:25.244661 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:25.244645    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret podName:fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7 nodeName:}" failed. No retries permitted until 2026-07-14 17:32:26.244630651 +0000 UTC m=+2.961024322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret") pod "global-pull-secret-syncer-fk7h5" (UID: "fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7") : object "kube-system"/"original-pull-secret" not registered
Jul 14 17:32:25.244661 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:25.244664    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb podName:ab91d593-f6ad-4a22-9d7f-e90a0dcc9787 nodeName:}" failed. No retries permitted until 2026-07-14 17:32:26.24465493 +0000 UTC m=+2.961048601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4xpb" (UniqueName: "kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb") pod "network-check-target-hv7kv" (UID: "ab91d593-f6ad-4a22-9d7f-e90a0dcc9787") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:32:25.462910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.462880    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 17:32:25.676996 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.676719    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 17:27:24 +0000 UTC" deadline="2027-12-31 02:53:28.384887842 +0000 UTC"
Jul 14 17:32:25.676996 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.676784    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12825h21m2.708108671s"
Jul 14 17:32:25.780637 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.780007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:25.780637 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:25.780137    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:25.784490 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.784402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-786wn" event={"ID":"52e52a4c-3f78-40f6-8d9e-963ff65c0f86","Type":"ContainerStarted","Data":"1dee10b3d7c16548f559c653c2a4d3124436eb1ffd6beb012c55118b84f8c3b4"}
Jul 14 17:32:25.788049 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.788010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rcnm5" event={"ID":"f715504c-a483-4450-838c-06e8d0c57b4c","Type":"ContainerStarted","Data":"071d60a6aa9503a0a160097bd0b70be42461d7efaefc57b343ae19c3681b89e9"}
Jul 14 17:32:25.789944 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.789916    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fdndd" event={"ID":"5afed38b-3690-4505-9749-3d026e206a4f","Type":"ContainerStarted","Data":"fe0e684323f9564b274d01f288b772bc107b3df9850eba9a67c30333dd377ff1"}
Jul 14 17:32:25.793390 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.793363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lftjp" event={"ID":"975b9e78-ce91-4d64-8ac5-0996dbdeab4a","Type":"ContainerStarted","Data":"b7694574c968a7ce39341a6ca4a26a73ecaaeb003ee796490c446291e7e8ee8d"}
Jul 14 17:32:25.794997 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.794968    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal" event={"ID":"e8a89385d90f63ff945cda7868eed5df","Type":"ContainerStarted","Data":"d84c7347ce9b15360928de01cbdb9aa5ecb2cecb4e2bbecd4e6cdbf9133417bc"}
Jul 14 17:32:25.802153 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.802082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2" event={"ID":"4349f767-b8be-4113-b166-607fdcd075ca","Type":"ContainerStarted","Data":"04c21c676607f476e33560308ad7e1b2bf43182ab679f78690a17c7c4be3ff55"}
Jul 14 17:32:25.803742 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.803675    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal" event={"ID":"99dd3a4d2ab0d3d55d4198e9e5c8c72d","Type":"ContainerStarted","Data":"1509f0197a10e57d6ab30c4ed6483dee9a41cbae92e4d0c00ac4a64d0f10f858"}
Jul 14 17:32:25.806070 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.806032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5" event={"ID":"e16a2359-cd0f-4b37-99d9-938bd35c13f7","Type":"ContainerStarted","Data":"26501c8e017afd111d0fef5d308774a4c6902654cbab040ed5767ae1c5b0203e"}
Jul 14 17:32:25.810327 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.810161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6xckn" event={"ID":"7792d2f3-ffbe-4009-8459-c38262ce3c30","Type":"ContainerStarted","Data":"c3272769b1d834d2b04c393f3bbfb799bd9fff63a264faefd17d88cb87fb449b"}
Jul 14 17:32:25.813950 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.813924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wfqp" event={"ID":"ab594791-63ca-446f-b89b-7e8c7724b7e7","Type":"ContainerStarted","Data":"eb4ab218b491530727f838dc3c23b7f27e096a230b5b6a8efc3c65987121c7df"}
Jul 14 17:32:25.818750 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:25.818723    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-554b4" event={"ID":"6f85b40b-bca9-4452-ac81-5786ca39e3e4","Type":"ContainerStarted","Data":"2496334e86a8ab70c54873cc86a1236761eadb4e88bb7d66666c9cac3dcbafbc"}
Jul 14 17:32:25.819251 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:25.819185    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d1573db16143027c8ffaf46a01ce12f2bcd89c15410c42ca54be4b3b1ddfdd6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d1573db16143027c8ffaf46a01ce12f2bcd89c15410c42ca54be4b3b1ddfdd6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5" podUID="e16a2359-cd0f-4b37-99d9-938bd35c13f7"
Jul 14 17:32:26.254836 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:26.254790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xpb\" (UniqueName: \"kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb\") pod \"network-check-target-hv7kv\" (UID: \"ab91d593-f6ad-4a22-9d7f-e90a0dcc9787\") " pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:26.255050 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:26.254863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:26.255050 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:26.254901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs\") pod \"network-metrics-daemon-x8dr4\" (UID: \"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d\") " pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:26.255164 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:26.255051    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:32:26.255164 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:26.255118    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs podName:535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d nodeName:}" failed. No retries permitted until 2026-07-14 17:32:28.255099189 +0000 UTC m=+4.971492864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs") pod "network-metrics-daemon-x8dr4" (UID: "535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:32:26.255604 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:26.255574    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 17:32:26.255712 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:26.255611    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 17:32:26.255712 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:26.255627    2578 projected.go:196] Error preparing data for projected volume kube-api-access-b4xpb for pod openshift-network-diagnostics/network-check-target-hv7kv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:32:26.255712 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:26.255700    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb podName:ab91d593-f6ad-4a22-9d7f-e90a0dcc9787 nodeName:}" failed. No retries permitted until 2026-07-14 17:32:28.255658596 +0000 UTC m=+4.972052275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4xpb" (UniqueName: "kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb") pod "network-check-target-hv7kv" (UID: "ab91d593-f6ad-4a22-9d7f-e90a0dcc9787") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:32:26.255892 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:26.255772    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 17:32:26.255892 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:26.255808    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret podName:fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7 nodeName:}" failed. No retries permitted until 2026-07-14 17:32:28.25579621 +0000 UTC m=+4.972189898 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret") pod "global-pull-secret-syncer-fk7h5" (UID: "fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7") : object "kube-system"/"original-pull-secret" not registered
Jul 14 17:32:26.677419 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:26.677341    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 17:27:24 +0000 UTC" deadline="2028-04-19 13:39:04.644102608 +0000 UTC"
Jul 14 17:32:26.677419 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:26.677384    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15476h6m37.96672381s"
Jul 14 17:32:26.777041 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:26.777006    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:26.777269 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:26.777212    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:26.777717 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:26.777695    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:26.777815 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:26.777797    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:27.208742 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:27.208689    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 17:32:27.779269 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:27.779237    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:27.779710 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:27.779372    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:28.273564 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:28.273462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xpb\" (UniqueName: \"kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb\") pod \"network-check-target-hv7kv\" (UID: \"ab91d593-f6ad-4a22-9d7f-e90a0dcc9787\") " pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:28.273564 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:28.273521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:28.273564 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:28.273552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs\") pod \"network-metrics-daemon-x8dr4\" (UID: \"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d\") " pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:28.273833 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:28.273699    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:32:28.273833 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:28.273761    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs podName:535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d nodeName:}" failed. No retries permitted until 2026-07-14 17:32:32.27374108 +0000 UTC m=+8.990134759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs") pod "network-metrics-daemon-x8dr4" (UID: "535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:32:28.274119 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:28.274089    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 17:32:28.274119 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:28.274109    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 17:32:28.274119 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:28.274118    2578 projected.go:196] Error preparing data for projected volume kube-api-access-b4xpb for pod openshift-network-diagnostics/network-check-target-hv7kv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:32:28.274384 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:28.274149    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb podName:ab91d593-f6ad-4a22-9d7f-e90a0dcc9787 nodeName:}" failed. No retries permitted until 2026-07-14 17:32:32.274139695 +0000 UTC m=+8.990533365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4xpb" (UniqueName: "kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb") pod "network-check-target-hv7kv" (UID: "ab91d593-f6ad-4a22-9d7f-e90a0dcc9787") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:32:28.274384 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:28.274188    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 17:32:28.274384 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:28.274225    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret podName:fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7 nodeName:}" failed. No retries permitted until 2026-07-14 17:32:32.274217698 +0000 UTC m=+8.990611370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret") pod "global-pull-secret-syncer-fk7h5" (UID: "fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7") : object "kube-system"/"original-pull-secret" not registered
Jul 14 17:32:28.776543 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:28.776509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:28.776727 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:28.776660    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:28.776792 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:28.776737    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:28.776888 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:28.776852    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:29.777337 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:29.777305    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:29.777754 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:29.777434    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:30.777315 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:30.777275    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:30.777515 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:30.777275    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:30.777515 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:30.777430    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:30.777912 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:30.777542    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:31.777546 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:31.777263    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:31.777953 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:31.777608    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:32.303724 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:32.303684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xpb\" (UniqueName: \"kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb\") pod \"network-check-target-hv7kv\" (UID: \"ab91d593-f6ad-4a22-9d7f-e90a0dcc9787\") " pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:32.303896 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:32.303734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:32.303896 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:32.303766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs\") pod \"network-metrics-daemon-x8dr4\" (UID: \"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d\") " pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:32.303896 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:32.303859    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:32:32.303896 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:32.303862    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 17:32:32.304116 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:32.303862    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 17:32:32.304116 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:32.303931    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 17:32:32.304116 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:32.303945    2578 projected.go:196] Error preparing data for projected volume kube-api-access-b4xpb for pod openshift-network-diagnostics/network-check-target-hv7kv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:32:32.304116 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:32.303912    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs podName:535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d nodeName:}" failed. No retries permitted until 2026-07-14 17:32:40.303898704 +0000 UTC m=+17.020292374 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs") pod "network-metrics-daemon-x8dr4" (UID: "535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:32:32.304116 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:32.303997    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret podName:fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7 nodeName:}" failed. No retries permitted until 2026-07-14 17:32:40.30398104 +0000 UTC m=+17.020374712 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret") pod "global-pull-secret-syncer-fk7h5" (UID: "fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7") : object "kube-system"/"original-pull-secret" not registered
Jul 14 17:32:32.304116 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:32.304013    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb podName:ab91d593-f6ad-4a22-9d7f-e90a0dcc9787 nodeName:}" failed. No retries permitted until 2026-07-14 17:32:40.304003634 +0000 UTC m=+17.020397311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4xpb" (UniqueName: "kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb") pod "network-check-target-hv7kv" (UID: "ab91d593-f6ad-4a22-9d7f-e90a0dcc9787") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:32:32.776685 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:32.776600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:32.776849 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:32.776608    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:32.776849 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:32.776743    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:32.776849 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:32.776806    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:33.777710 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:33.777674    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:33.778104 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:33.777766    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:34.776834 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:34.776798    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:34.777015 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:34.776843    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:34.777015 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:34.776936    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:34.777126 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:34.777073    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:35.777629 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:35.777595    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:35.778065 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:35.777712    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:36.776946 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:36.776910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:36.777139 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:36.776924    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:36.777139 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:36.777006    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:36.777139 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:36.777125    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:37.776594 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:37.776565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:37.777049 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:37.776673    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:38.777107 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:38.777034    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:38.777469 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:38.777144    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:38.777469 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:38.777194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:38.777469 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:38.777308    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:39.777188 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.776984    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:39.777542 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:39.777231    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:39.859090 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.859056    2578 generic.go:356] "Generic (PLEG): container finished" podID="ab594791-63ca-446f-b89b-7e8c7724b7e7" containerID="5cbfa3758e3d2fc36deb412aed0848cddd4ba362be44fe326b786e066fab6035" exitCode=0
Jul 14 17:32:39.859275 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.859154    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wfqp" event={"ID":"ab594791-63ca-446f-b89b-7e8c7724b7e7","Type":"ContainerDied","Data":"5cbfa3758e3d2fc36deb412aed0848cddd4ba362be44fe326b786e066fab6035"}
Jul 14 17:32:39.861248 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.861223    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-554b4" event={"ID":"6f85b40b-bca9-4452-ac81-5786ca39e3e4","Type":"ContainerStarted","Data":"1f6e3529d91cb9e49d161249343b274a7ed9a21a2dae52d16a863095ed0b0ca9"}
Jul 14 17:32:39.862745 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.862719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-786wn" event={"ID":"52e52a4c-3f78-40f6-8d9e-963ff65c0f86","Type":"ContainerStarted","Data":"28728e2efc129a1aa4cb4400189efc1ff73ea8c04c4e3305eac26a65b49f4a33"}
Jul 14 17:32:39.865732 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.865551    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rcnm5" event={"ID":"f715504c-a483-4450-838c-06e8d0c57b4c","Type":"ContainerStarted","Data":"8acbcea1be4387c169f8c21b85a2a9404b1aa5ba8e97fa6e372b0a50f5f4d57b"}
Jul 14 17:32:39.867105 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.867077    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fdndd" event={"ID":"5afed38b-3690-4505-9749-3d026e206a4f","Type":"ContainerStarted","Data":"e7ead946a8f413f7d1c9589d55f64d4724c8242133840ed466240cce887fcf7c"}
Jul 14 17:32:39.868542 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.868518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lftjp" event={"ID":"975b9e78-ce91-4d64-8ac5-0996dbdeab4a","Type":"ContainerStarted","Data":"e1535e042fc0d541ec0d21ffe5ab39698b3c4460be5838e57a1e2f2200f15f9d"}
Jul 14 17:32:39.870787 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.870750    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal" event={"ID":"e8a89385d90f63ff945cda7868eed5df","Type":"ContainerStarted","Data":"b7f45f321b56e971782abd60d68af07300918cfd44fe0259874f6e955610f8c2"}
Jul 14 17:32:39.872395 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.872374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2" event={"ID":"4349f767-b8be-4113-b166-607fdcd075ca","Type":"ContainerStarted","Data":"1e54c9faeb47bd46a672701539fd4dd7d9a15c915d4ef3e78ee410eb79b977a4"}
Jul 14 17:32:39.873997 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.873974    2578 generic.go:356] "Generic (PLEG): container finished" podID="99dd3a4d2ab0d3d55d4198e9e5c8c72d" containerID="1ebc4cd09342601646e70a5155e64d28e856f4301be67160434e49a00ca6dce7" exitCode=0
Jul 14 17:32:39.874092 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.874006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal" event={"ID":"99dd3a4d2ab0d3d55d4198e9e5c8c72d","Type":"ContainerDied","Data":"1ebc4cd09342601646e70a5155e64d28e856f4301be67160434e49a00ca6dce7"}
Jul 14 17:32:39.917619 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.917567    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal" podStartSLOduration=15.917546677 podStartE2EDuration="15.917546677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:32:39.901833155 +0000 UTC m=+16.618226849" watchObservedRunningTime="2026-07-14 17:32:39.917546677 +0000 UTC m=+16.633940370"
Jul 14 17:32:39.917885 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.917858    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-786wn" podStartSLOduration=2.098815917 podStartE2EDuration="15.917851581s" totalImagesPullingTime="13.819035664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:32:39.917359103 +0000 UTC m=+16.633752796" watchObservedRunningTime="2026-07-14 17:32:39.917851581 +0000 UTC m=+16.634245275"
Jul 14 17:32:39.952454 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.952395    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-554b4" podStartSLOduration=2.140966904 podStartE2EDuration="15.952375213s" totalImagesPullingTime="13.811408309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:32:39.932527968 +0000 UTC m=+16.648921660" watchObservedRunningTime="2026-07-14 17:32:39.952375213 +0000 UTC m=+16.668768907"
Jul 14 17:32:39.952723 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.952694    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lftjp" podStartSLOduration=2.128423787 podStartE2EDuration="15.952680236s" totalImagesPullingTime="13.824256449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:32:39.950707917 +0000 UTC m=+16.667101609" watchObservedRunningTime="2026-07-14 17:32:39.952680236 +0000 UTC m=+16.669073928"
Jul 14 17:32:39.982583 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:39.982523    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fdndd" podStartSLOduration=2.110324604 podStartE2EDuration="15.982501096s" totalImagesPullingTime="13.872176492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:32:39.981608069 +0000 UTC m=+16.698001760" watchObservedRunningTime="2026-07-14 17:32:39.982501096 +0000 UTC m=+16.698894789"
Jul 14 17:32:40.000442 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.000378    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rcnm5" podStartSLOduration=2.027951134 podStartE2EDuration="16.000358849s" totalImagesPullingTime="13.972407715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:32:40.000322249 +0000 UTC m=+16.716715969" watchObservedRunningTime="2026-07-14 17:32:40.000358849 +0000 UTC m=+16.716752541"
Jul 14 17:32:40.059403 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.059315    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fdndd"
Jul 14 17:32:40.060046 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.060023    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fdndd"
Jul 14 17:32:40.105771 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.105736    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:40.177823 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.177784    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:40.292878 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.292855    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 14 17:32:40.363599 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.363512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xpb\" (UniqueName: \"kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb\") pod \"network-check-target-hv7kv\" (UID: \"ab91d593-f6ad-4a22-9d7f-e90a0dcc9787\") " pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:40.363599 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.363560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:40.363599 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.363582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs\") pod \"network-metrics-daemon-x8dr4\" (UID: \"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d\") " pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:40.363823 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:40.363692    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 17:32:40.363823 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:40.363714    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 17:32:40.363823 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:40.363723    2578 projected.go:196] Error preparing data for projected volume kube-api-access-b4xpb for pod openshift-network-diagnostics/network-check-target-hv7kv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:32:40.363823 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:40.363769    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb podName:ab91d593-f6ad-4a22-9d7f-e90a0dcc9787 nodeName:}" failed. No retries permitted until 2026-07-14 17:32:56.363755899 +0000 UTC m=+33.080149583 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4xpb" (UniqueName: "kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb") pod "network-check-target-hv7kv" (UID: "ab91d593-f6ad-4a22-9d7f-e90a0dcc9787") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:32:40.363823 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:40.363692    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 17:32:40.364002 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:40.363832    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret podName:fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7 nodeName:}" failed. No retries permitted until 2026-07-14 17:32:56.363815005 +0000 UTC m=+33.080208694 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret") pod "global-pull-secret-syncer-fk7h5" (UID: "fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7") : object "kube-system"/"original-pull-secret" not registered
Jul 14 17:32:40.364002 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:40.363691    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:32:40.364002 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:40.363878    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs podName:535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d nodeName:}" failed. No retries permitted until 2026-07-14 17:32:56.363867328 +0000 UTC m=+33.080261025 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs") pod "network-metrics-daemon-x8dr4" (UID: "535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:32:40.699990 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.699836    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T17:32:40.292875765Z","UUID":"896cb798-b59f-4dbf-8e44-e2f311af8514","Handler":null,"Name":"","Endpoint":""}
Jul 14 17:32:40.701515 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.701492    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 14 17:32:40.701633 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.701530    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 14 17:32:40.777505 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.777472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:40.778253 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:40.777584    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:40.778253 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.777602    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:40.778253 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:40.777903    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:40.877799 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.877757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6xckn" event={"ID":"7792d2f3-ffbe-4009-8459-c38262ce3c30","Type":"ContainerStarted","Data":"26a3b29944805fdde748db2c5d0680df0635c045b3e9c1d62ba2cf05df14c402"}
Jul 14 17:32:40.879635 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.879608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2" event={"ID":"4349f767-b8be-4113-b166-607fdcd075ca","Type":"ContainerStarted","Data":"bb29412dd558196301d1247767bfec1b1fb78be38cd3a9820c6bb465be8e796b"}
Jul 14 17:32:40.881947 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.881550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal" event={"ID":"99dd3a4d2ab0d3d55d4198e9e5c8c72d","Type":"ContainerStarted","Data":"fd805cc4ee30af104009d71330d8551b3fa60163adbb8728d69a7f74448260b4"}
Jul 14 17:32:40.894112 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.894063    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6xckn" podStartSLOduration=3.1311235809999998 podStartE2EDuration="16.894049722s" totalImagesPullingTime="13.762926141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:32:40.893670958 +0000 UTC m=+17.610064652" watchObservedRunningTime="2026-07-14 17:32:40.894049722 +0000 UTC m=+17.610443413"
Jul 14 17:32:40.910637 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:40.910586    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal" podStartSLOduration=16.910572123 podStartE2EDuration="16.910572123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:32:40.909930342 +0000 UTC m=+17.626324034" watchObservedRunningTime="2026-07-14 17:32:40.910572123 +0000 UTC m=+17.626965814"
Jul 14 17:32:41.023738 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:41.023595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:41.089136 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:41.089084    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:41.778050 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:41.777484    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:41.778050 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:41.777612    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:41.889086 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:41.889051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2" event={"ID":"4349f767-b8be-4113-b166-607fdcd075ca","Type":"ContainerStarted","Data":"ac726177db6e6292388ed1b8371268de9bf36145d666e166b403815a024427b0"}
Jul 14 17:32:41.889518 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:41.889498    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 14 17:32:41.914360 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:41.914309    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-492s2" podStartSLOduration=2.223664526 podStartE2EDuration="17.914290076s" totalImagesPullingTime="15.69062555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:32:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:32:41.913627388 +0000 UTC m=+18.630021081" watchObservedRunningTime="2026-07-14 17:32:41.914290076 +0000 UTC m=+18.630683771"
Jul 14 17:32:42.776906 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:42.776663    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:42.777070 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:42.776663    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:42.777070 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:42.776995    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:42.777193 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:42.777109    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:43.777584 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:43.777552    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:43.778063 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:43.777668    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:44.776518 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:44.776485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:44.776669 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:44.776633    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:44.777003 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:44.776987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:44.777079 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:44.777062    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:45.096495 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:45.096406    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:45.196336 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:45.196309    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:45.609190 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:45.609150    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:45.709072 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:45.709043    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:45.777200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:45.777163    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:45.777390 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:45.777313    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:45.850860 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:45.850825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:45.899375 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:45.899282    2578 generic.go:356] "Generic (PLEG): container finished" podID="ab594791-63ca-446f-b89b-7e8c7724b7e7" containerID="793b59e87ed4b8ee7b0fd9f21efe006a46ca930b76d6a4cd70ba2c6b81e47fc3" exitCode=0
Jul 14 17:32:45.899375 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:45.899348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wfqp" event={"ID":"ab594791-63ca-446f-b89b-7e8c7724b7e7","Type":"ContainerDied","Data":"793b59e87ed4b8ee7b0fd9f21efe006a46ca930b76d6a4cd70ba2c6b81e47fc3"}
Jul 14 17:32:45.932996 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:45.932963    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:46.048752 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:46.048722    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:46.145076 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:46.145046    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:46.296507 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:46.296429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:46.379880 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:46.379852    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:46.755126 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:46.755056    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:46.776649 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:46.776627    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:46.776773 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:46.776750    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:46.776836 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:46.776626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:46.776884 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:46.776841    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:46.884671 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:46.884639    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:46.904485 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:46.904448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wfqp" event={"ID":"ab594791-63ca-446f-b89b-7e8c7724b7e7","Type":"ContainerStarted","Data":"28dd19852a4f7aef92f9758ec21f0a9e773c83ede22d2858386820d6f636d8ca"}
Jul 14 17:32:46.985826 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:46.985795    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:47.347434 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:47.347399    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:47.444043 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:47.444011    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-786wn_52e52a4c-3f78-40f6-8d9e-963ff65c0f86/dns-node-resolver/0.log"
Jul 14 17:32:47.446241 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:47.446196    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:47.578188 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:47.578134    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:47.776976 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:47.776898    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:47.777134 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:47.777028    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:47.909159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:47.909124    2578 generic.go:356] "Generic (PLEG): container finished" podID="ab594791-63ca-446f-b89b-7e8c7724b7e7" containerID="28dd19852a4f7aef92f9758ec21f0a9e773c83ede22d2858386820d6f636d8ca" exitCode=0
Jul 14 17:32:47.909335 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:47.909189    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wfqp" event={"ID":"ab594791-63ca-446f-b89b-7e8c7724b7e7","Type":"ContainerDied","Data":"28dd19852a4f7aef92f9758ec21f0a9e773c83ede22d2858386820d6f636d8ca"}
Jul 14 17:32:48.012816 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:48.012771    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:48.080035 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:48.079947    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:48.445085 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:48.445010    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-554b4_6f85b40b-bca9-4452-ac81-5786ca39e3e4/node-ca/0.log"
Jul 14 17:32:48.776618 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:48.776546    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:48.776618 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:48.776576    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:48.776825 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:48.776687    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:48.776867 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:48.776819    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:48.984097 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:48.984054    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:49.777221 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:49.777170    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:49.777628 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:49.777325    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:50.018737 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:50.018697    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:50.142557 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:50.142465    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:50.241376 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:50.241311    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:50.340772 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:50.340743    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:50.421236 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:50.421137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:50.631133 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:50.631096    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:50.747654 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:50.747557    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:50.776768 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:50.776724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:50.776768 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:50.776746    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:50.776986 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:50.776868    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:50.777057 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:50.777023    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:51.718322 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:51.718246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:51.777521 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:51.777490    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:51.777684 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:51.777624    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:52.435059 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:52.435025    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:52.534599 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:52.534567    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:52.635833 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:52.635804    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:52.777229 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:52.777191    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:52.777579 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:52.777297    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:52.777579 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:52.777193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:52.777579 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:52.777366    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:52.915903 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:52.915699    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:52.921383 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:52.921361    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 17:32:52.921777 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:52.921757    2578 generic.go:356] "Generic (PLEG): container finished" podID="e16a2359-cd0f-4b37-99d9-938bd35c13f7" containerID="3931e6b5a867eb9f8f60ad2178f329e114325d83e0ed74b9f1773aab009f7257" exitCode=1
Jul 14 17:32:52.921863 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:52.921810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5" event={"ID":"e16a2359-cd0f-4b37-99d9-938bd35c13f7","Type":"ContainerDied","Data":"3931e6b5a867eb9f8f60ad2178f329e114325d83e0ed74b9f1773aab009f7257"}
Jul 14 17:32:52.921900 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:52.921867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5" event={"ID":"e16a2359-cd0f-4b37-99d9-938bd35c13f7","Type":"ContainerStarted","Data":"06c4e9726b076a30eb1ed6352ca8e932dd0bce1c23a27cc166839eef47b9d3ec"}
Jul 14 17:32:53.183530 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:53.183497    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:53.312765 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:53.312729    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fdndd"
Jul 14 17:32:53.313086 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:53.312868    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 14 17:32:53.313295 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:53.313275    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fdndd"
Jul 14 17:32:53.715825 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:53.715786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:53.786515 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:53.786431    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:53.787105 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:53.786514    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:53.788612 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:53.788587    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:53.926963 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:53.926927    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 17:32:53.927346 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:53.927324    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5" event={"ID":"e16a2359-cd0f-4b37-99d9-938bd35c13f7","Type":"ContainerStarted","Data":"447ab5706cde5c67cd1bb145eb8920977a2bda68012411623b4289229d3574e7"}
Jul 14 17:32:53.927430 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:53.927408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5" event={"ID":"e16a2359-cd0f-4b37-99d9-938bd35c13f7","Type":"ContainerStarted","Data":"132454a968a3e6ff06d0e30d89223b97e589e61a48d4fadaadd0f977e94e9cb9"}
Jul 14 17:32:53.927536 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:53.927467    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5" event={"ID":"e16a2359-cd0f-4b37-99d9-938bd35c13f7","Type":"ContainerStarted","Data":"5d41de876772daa17996ee346b68b8d12d050b74d11c79aa21cc5d7399c36552"}
Jul 14 17:32:53.927536 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:53.927516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5" event={"ID":"e16a2359-cd0f-4b37-99d9-938bd35c13f7","Type":"ContainerStarted","Data":"4ac66e3676ba9729b9febef02f5749c31b05929c4bd61fa92bbce28ea2f727da"}
Jul 14 17:32:53.929664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:53.929643    2578 generic.go:356] "Generic (PLEG): container finished" podID="ab594791-63ca-446f-b89b-7e8c7724b7e7" containerID="10ec0c9b070b52cf8d7b93065da74039f88dbd988e198e830e4b1ac7c987bbed" exitCode=0
Jul 14 17:32:53.929742 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:53.929720    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wfqp" event={"ID":"ab594791-63ca-446f-b89b-7e8c7724b7e7","Type":"ContainerDied","Data":"10ec0c9b070b52cf8d7b93065da74039f88dbd988e198e830e4b1ac7c987bbed"}
Jul 14 17:32:54.365437 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:54.365400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:54.428267 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:54.428236    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:54.516075 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:54.516039    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:54.616948 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:54.616867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:54.777190 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:54.777161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:54.777190 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:54.777180    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:54.777405 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:54.777300    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:54.777453 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:54.777382    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:54.913473 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:54.913357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:55.528561 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:55.528534    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:55.777593 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:55.777511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:55.777758 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:55.777611    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:55.938141 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:55.938114    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 17:32:55.938579 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:55.938530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5" event={"ID":"e16a2359-cd0f-4b37-99d9-938bd35c13f7","Type":"ContainerStarted","Data":"8e464f3379dd82429db1b624a7c7d936ca6d853dcb64e9ef129a5734d1bb4ccd"}
Jul 14 17:32:56.136635 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:56.136547    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:56.237164 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:56.237117    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:56.338301 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:56.338267    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:56.385316 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:56.385286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xpb\" (UniqueName: \"kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb\") pod \"network-check-target-hv7kv\" (UID: \"ab91d593-f6ad-4a22-9d7f-e90a0dcc9787\") " pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:56.385474 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:56.385327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:56.385474 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:56.385350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs\") pod \"network-metrics-daemon-x8dr4\" (UID: \"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d\") " pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:56.385474 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:56.385445    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:32:56.385474 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:56.385458    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 17:32:56.385474 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:56.385469    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 17:32:56.385678 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:56.385489    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 17:32:56.385678 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:56.385492    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs podName:535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d nodeName:}" failed. No retries permitted until 2026-07-14 17:33:28.385477272 +0000 UTC m=+65.101870942 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs") pod "network-metrics-daemon-x8dr4" (UID: "535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:32:56.385678 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:56.385499    2578 projected.go:196] Error preparing data for projected volume kube-api-access-b4xpb for pod openshift-network-diagnostics/network-check-target-hv7kv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:32:56.385678 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:56.385516    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret podName:fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7 nodeName:}" failed. No retries permitted until 2026-07-14 17:33:28.385498105 +0000 UTC m=+65.101891779 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret") pod "global-pull-secret-syncer-fk7h5" (UID: "fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7") : object "kube-system"/"original-pull-secret" not registered
Jul 14 17:32:56.385678 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:56.385532    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb podName:ab91d593-f6ad-4a22-9d7f-e90a0dcc9787 nodeName:}" failed. No retries permitted until 2026-07-14 17:33:28.385522803 +0000 UTC m=+65.101916473 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4xpb" (UniqueName: "kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb") pod "network-check-target-hv7kv" (UID: "ab91d593-f6ad-4a22-9d7f-e90a0dcc9787") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:32:56.776837 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:56.776801    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:56.776837 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:56.776837    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:56.777069 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:56.776933    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:56.777125 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:56.777071    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:57.245991 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:57.245907    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:57.345783 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:57.345746    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:57.777185 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:57.777105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:57.777364 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:57.777239    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:58.112647 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.112620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:58.356574 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.356156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:58.714739 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.714664    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:58.776738 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.776715    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:32:58.776892 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.776715    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:32:58.776892 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:58.776822    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:32:58.776993 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:58.776901    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:32:58.926410 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.926378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:58.947772 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.947747    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 17:32:58.948107 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.948086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5" event={"ID":"e16a2359-cd0f-4b37-99d9-938bd35c13f7","Type":"ContainerStarted","Data":"683cf8f77dbb5f89d61fb390326c5bc29c5703461c75dc2a38b617a005ef9679"}
Jul 14 17:32:58.948462 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.948327    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:58.948462 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.948383    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:58.948462 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.948418    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:58.948695 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.948491    2578 scope.go:117] "RemoveContainer" containerID="3931e6b5a867eb9f8f60ad2178f329e114325d83e0ed74b9f1773aab009f7257"
Jul 14 17:32:58.952199 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.951786    2578 generic.go:356] "Generic (PLEG): container finished" podID="ab594791-63ca-446f-b89b-7e8c7724b7e7" containerID="5a5e93145ca6d4547ebbedea245b738b84488147185547f0b5a9ffcf37714905" exitCode=0
Jul 14 17:32:58.952199 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.951831    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wfqp" event={"ID":"ab594791-63ca-446f-b89b-7e8c7724b7e7","Type":"ContainerDied","Data":"5a5e93145ca6d4547ebbedea245b738b84488147185547f0b5a9ffcf37714905"}
Jul 14 17:32:58.967162 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.967103    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:58.967277 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:58.967261    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:32:59.322368 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:59.322272    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:32:59.777434 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:59.777356    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:32:59.777808 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:32:59.777455    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:32:59.957304 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:59.957270    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 17:32:59.957697 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:59.957659    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5" event={"ID":"e16a2359-cd0f-4b37-99d9-938bd35c13f7","Type":"ContainerStarted","Data":"08cabac541c09db2e6cf9eb8bd75af72a2c52e48e9bae18d7664337a419ee296"}
Jul 14 17:32:59.960676 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:59.960653    2578 generic.go:356] "Generic (PLEG): container finished" podID="ab594791-63ca-446f-b89b-7e8c7724b7e7" containerID="85d28443490d3ea4dce42af588a9106cf0032a4a25db84d3c3b6eb0dbe19b415" exitCode=0
Jul 14 17:32:59.960779 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:32:59.960694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wfqp" event={"ID":"ab594791-63ca-446f-b89b-7e8c7724b7e7","Type":"ContainerDied","Data":"85d28443490d3ea4dce42af588a9106cf0032a4a25db84d3c3b6eb0dbe19b415"}
Jul 14 17:33:00.022925 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:00.022874    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5" podStartSLOduration=8.658328189 podStartE2EDuration="36.022859408s" totalImagesPullingTime="27.364531219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=2 observedRunningTime="2026-07-14 17:33:00.022623719 +0000 UTC m=+36.739017424" watchObservedRunningTime="2026-07-14 17:33:00.022859408 +0000 UTC m=+36.739253100"
Jul 14 17:33:00.133514 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:00.133479    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:00.314278 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:00.314142    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:00.595324 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:00.595225    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:00.698180 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:00.698150    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:00.758198 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:00.758168    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:00.777191 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:00.777166    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:33:00.777318 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:00.777281    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:33:00.777374 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:00.777345    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:33:00.777476 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:00.777458    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:33:00.966662 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:00.966627    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wfqp" event={"ID":"ab594791-63ca-446f-b89b-7e8c7724b7e7","Type":"ContainerStarted","Data":"2657cfe29b550c7b989687972181b83c0da52f26f3d1d9b55c5564d309919d44"}
Jul 14 17:33:00.991191 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:00.991141    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6wfqp" podStartSLOduration=7.740618628 podStartE2EDuration="36.99112532s" totalImagesPullingTime="29.250506692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:32:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:33:00.990279436 +0000 UTC m=+37.706673128" watchObservedRunningTime="2026-07-14 17:33:00.99112532 +0000 UTC m=+37.707519012"
Jul 14 17:33:01.777561 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:01.777533    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:33:01.777893 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:01.777627    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:33:02.359667 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:02.359629    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:02.777289 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:02.777260    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:33:02.777511 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:02.777363    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:33:02.777511 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:02.777415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:33:02.777511 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:02.777503    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:33:02.979307 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:02.979276    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:03.778148 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:03.778113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:33:03.778324 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:03.778217    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:33:04.273329 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:04.273286    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:04.392068 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:04.392036    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:04.777149 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:04.777121    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:33:04.777333 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:04.777121    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:33:04.777333 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:04.777254    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:33:04.777333 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:04.777300    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:33:05.777457 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:05.777428    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:33:05.777813 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:05.777529    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:33:06.776844 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:06.776812    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:33:06.777042 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:06.776811    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:33:06.777042 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:06.776915    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:33:06.777042 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:06.777009    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:33:07.777069 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:07.777038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:33:07.777575 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:07.777138    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:33:08.776951 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:08.776916    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:33:08.777117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:08.776920    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:33:08.777117 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:08.777015    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:33:08.777117 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:08.777105    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:33:09.780337 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:09.780311    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:33:09.780701 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:09.780411    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:33:09.966930 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:09.966889    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:10.065468 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:10.065383    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:10.777167 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:10.777135    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:33:10.777358 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:10.777142    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:33:10.777358 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:10.777249    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:33:10.777358 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:10.777313    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:33:11.188377 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:11.188342    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:11.273184 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:11.273142    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:11.654150 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:11.654069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:11.744171 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:11.744141    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:11.777233 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:11.777216    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:33:11.777320 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:11.777303    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:33:11.990950 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:11.990911    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:12.103090 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:12.103056    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:12.207842 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:12.207670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:12.311087 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:12.310995    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:12.402707 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:12.402658    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:12.638772 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:12.638680    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:12.777191 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:12.777166    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:33:12.777367 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:12.777172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:33:12.777367 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:12.777269    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:33:12.777367 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:12.777337    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:33:13.300034 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:13.300005    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:13.374417 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:13.374385    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fk7h5"]
Jul 14 17:33:13.374594 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:13.374505    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:33:13.374655 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:13.374620    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:33:13.377794 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:13.377768    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x8dr4"]
Jul 14 17:33:13.377928 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:13.377917    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:33:13.378058 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:13.378026    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:33:13.378498 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:13.378471    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hv7kv"]
Jul 14 17:33:13.378634 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:13.378561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:33:13.378700 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:13.378653    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:33:13.670958 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:13.670923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:13.766732 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:13.766703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:14.732469 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:14.732436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:14.777424 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:14.777383    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:33:14.777424 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:14.777432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:33:14.777616 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:14.777432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:33:14.777616 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:14.777495    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:33:14.777616 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:14.777583    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:33:14.777735 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:14.777650    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:33:14.803498 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:14.803466    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:16.776809 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:16.776765    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:33:16.777235 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:16.776766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:33:16.777235 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:16.776983    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8dr4" podUID="535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d"
Jul 14 17:33:16.777235 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:16.776881    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv7kv" podUID="ab91d593-f6ad-4a22-9d7f-e90a0dcc9787"
Jul 14 17:33:16.777235 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:16.776786    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:33:16.777235 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:16.777081    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fk7h5" podUID="fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7"
Jul 14 17:33:17.105035 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.105009    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeReady"
Jul 14 17:33:17.105228 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.105130    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 14 17:33:17.193691 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.193664    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f6b6674b9-4lr8w"]
Jul 14 17:33:17.195628 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.195608    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.207358 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.207340    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 14 17:33:17.207754 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.207737    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 14 17:33:17.212429 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.212414    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 14 17:33:17.213376 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.213361    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vhc85\""
Jul 14 17:33:17.224109 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.223941    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 14 17:33:17.232388 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.232369    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tcqk5"]
Jul 14 17:33:17.234089 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.234076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tcqk5"
Jul 14 17:33:17.238626 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.238610    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wwng2\""
Jul 14 17:33:17.238902 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.238884    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 14 17:33:17.239020 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.238908    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 14 17:33:17.239176 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.239163    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 14 17:33:17.243528 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.243508    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f6b6674b9-4lr8w"]
Jul 14 17:33:17.250475 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.250458    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-f76mp"]
Jul 14 17:33:17.252076 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.252063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:17.253888 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.253869    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tcqk5"]
Jul 14 17:33:17.254458 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.254439    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 14 17:33:17.254555 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.254493    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wv6hh\""
Jul 14 17:33:17.254555 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.254504    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 14 17:33:17.273033 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.273012    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f76mp"]
Jul 14 17:33:17.312940 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.312908    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s48hg"]
Jul 14 17:33:17.314901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.314884    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.318187 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.318167    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tw9bp\""
Jul 14 17:33:17.318298 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.318227    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 14 17:33:17.318298 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.318266    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 14 17:33:17.318648 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.318635    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 14 17:33:17.321573 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.321553    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 14 17:33:17.337172 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.337148    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s48hg"]
Jul 14 17:33:17.345983 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.345962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2317c923-3e9d-4bee-b3fe-760df78d2176-cert\") pod \"ingress-canary-tcqk5\" (UID: \"2317c923-3e9d-4bee-b3fe-760df78d2176\") " pod="openshift-ingress-canary/ingress-canary-tcqk5"
Jul 14 17:33:17.346090 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.345987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89dbd5c1-c9c4-4182-83e4-b1c086db1654-metrics-tls\") pod \"dns-default-f76mp\" (UID: \"89dbd5c1-c9c4-4182-83e4-b1c086db1654\") " pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:17.346090 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.346004    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-installation-pull-secrets\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.346090 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.346021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxz5d\" (UniqueName: \"kubernetes.io/projected/2317c923-3e9d-4bee-b3fe-760df78d2176-kube-api-access-fxz5d\") pod \"ingress-canary-tcqk5\" (UID: \"2317c923-3e9d-4bee-b3fe-760df78d2176\") " pod="openshift-ingress-canary/ingress-canary-tcqk5"
Jul 14 17:33:17.346090 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.346037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-image-registry-private-configuration\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.346324 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.346091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-trusted-ca\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.346324 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.346199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-ca-trust-extracted\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.346324 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.346258    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2s6j\" (UniqueName: \"kubernetes.io/projected/89dbd5c1-c9c4-4182-83e4-b1c086db1654-kube-api-access-k2s6j\") pod \"dns-default-f76mp\" (UID: \"89dbd5c1-c9c4-4182-83e4-b1c086db1654\") " pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:17.346324 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.346285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-registry-certificates\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.346324 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.346315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-bound-sa-token\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.346538 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.346350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-registry-tls\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.346538 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.346382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89dbd5c1-c9c4-4182-83e4-b1c086db1654-config-volume\") pod \"dns-default-f76mp\" (UID: \"89dbd5c1-c9c4-4182-83e4-b1c086db1654\") " pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:17.346538 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.346409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89dbd5c1-c9c4-4182-83e4-b1c086db1654-tmp-dir\") pod \"dns-default-f76mp\" (UID: \"89dbd5c1-c9c4-4182-83e4-b1c086db1654\") " pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:17.346538 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.346431    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8vqz\" (UniqueName: \"kubernetes.io/projected/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-kube-api-access-h8vqz\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.447146 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2317c923-3e9d-4bee-b3fe-760df78d2176-cert\") pod \"ingress-canary-tcqk5\" (UID: \"2317c923-3e9d-4bee-b3fe-760df78d2176\") " pod="openshift-ingress-canary/ingress-canary-tcqk5"
Jul 14 17:33:17.447146 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89dbd5c1-c9c4-4182-83e4-b1c086db1654-metrics-tls\") pod \"dns-default-f76mp\" (UID: \"89dbd5c1-c9c4-4182-83e4-b1c086db1654\") " pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:17.447403 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-installation-pull-secrets\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.447403 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4xc4\" (UniqueName: \"kubernetes.io/projected/4186b2ec-a392-40d8-b06d-8be76fa35bbe-kube-api-access-v4xc4\") pod \"insights-runtime-extractor-s48hg\" (UID: \"4186b2ec-a392-40d8-b06d-8be76fa35bbe\") " pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.447403 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxz5d\" (UniqueName: \"kubernetes.io/projected/2317c923-3e9d-4bee-b3fe-760df78d2176-kube-api-access-fxz5d\") pod \"ingress-canary-tcqk5\" (UID: \"2317c923-3e9d-4bee-b3fe-760df78d2176\") " pod="openshift-ingress-canary/ingress-canary-tcqk5"
Jul 14 17:33:17.447403 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-image-registry-private-configuration\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.447403 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-trusted-ca\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.447403 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4186b2ec-a392-40d8-b06d-8be76fa35bbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s48hg\" (UID: \"4186b2ec-a392-40d8-b06d-8be76fa35bbe\") " pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.447403 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-ca-trust-extracted\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.447745 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4186b2ec-a392-40d8-b06d-8be76fa35bbe-data-volume\") pod \"insights-runtime-extractor-s48hg\" (UID: \"4186b2ec-a392-40d8-b06d-8be76fa35bbe\") " pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.447745 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2s6j\" (UniqueName: \"kubernetes.io/projected/89dbd5c1-c9c4-4182-83e4-b1c086db1654-kube-api-access-k2s6j\") pod \"dns-default-f76mp\" (UID: \"89dbd5c1-c9c4-4182-83e4-b1c086db1654\") " pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:17.447745 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-registry-certificates\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.447745 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-bound-sa-token\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.447745 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-registry-tls\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.447745 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89dbd5c1-c9c4-4182-83e4-b1c086db1654-config-volume\") pod \"dns-default-f76mp\" (UID: \"89dbd5c1-c9c4-4182-83e4-b1c086db1654\") " pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:17.447745 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89dbd5c1-c9c4-4182-83e4-b1c086db1654-tmp-dir\") pod \"dns-default-f76mp\" (UID: \"89dbd5c1-c9c4-4182-83e4-b1c086db1654\") " pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:17.447745 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8vqz\" (UniqueName: \"kubernetes.io/projected/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-kube-api-access-h8vqz\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.447745 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4186b2ec-a392-40d8-b06d-8be76fa35bbe-crio-socket\") pod \"insights-runtime-extractor-s48hg\" (UID: \"4186b2ec-a392-40d8-b06d-8be76fa35bbe\") " pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.447745 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4186b2ec-a392-40d8-b06d-8be76fa35bbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s48hg\" (UID: \"4186b2ec-a392-40d8-b06d-8be76fa35bbe\") " pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.448190 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.447770    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-ca-trust-extracted\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.448190 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.448098    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89dbd5c1-c9c4-4182-83e4-b1c086db1654-tmp-dir\") pod \"dns-default-f76mp\" (UID: \"89dbd5c1-c9c4-4182-83e4-b1c086db1654\") " pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:17.448530 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.448508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-trusted-ca\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.448760 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.448736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-registry-certificates\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.448866 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.448841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89dbd5c1-c9c4-4182-83e4-b1c086db1654-config-volume\") pod \"dns-default-f76mp\" (UID: \"89dbd5c1-c9c4-4182-83e4-b1c086db1654\") " pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:17.451990 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.451966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89dbd5c1-c9c4-4182-83e4-b1c086db1654-metrics-tls\") pod \"dns-default-f76mp\" (UID: \"89dbd5c1-c9c4-4182-83e4-b1c086db1654\") " pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:17.451990 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.451979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-installation-pull-secrets\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.452107 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.451990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-image-registry-private-configuration\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.452107 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.452037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2317c923-3e9d-4bee-b3fe-760df78d2176-cert\") pod \"ingress-canary-tcqk5\" (UID: \"2317c923-3e9d-4bee-b3fe-760df78d2176\") " pod="openshift-ingress-canary/ingress-canary-tcqk5"
Jul 14 17:33:17.452107 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.452078    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-registry-tls\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.456516 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.456490    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8vqz\" (UniqueName: \"kubernetes.io/projected/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-kube-api-access-h8vqz\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.457567 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.457542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6f3fcfd-03da-490d-aef8-06ef2c6875bb-bound-sa-token\") pod \"image-registry-f6b6674b9-4lr8w\" (UID: \"a6f3fcfd-03da-490d-aef8-06ef2c6875bb\") " pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.457703 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.457688    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxz5d\" (UniqueName: \"kubernetes.io/projected/2317c923-3e9d-4bee-b3fe-760df78d2176-kube-api-access-fxz5d\") pod \"ingress-canary-tcqk5\" (UID: \"2317c923-3e9d-4bee-b3fe-760df78d2176\") " pod="openshift-ingress-canary/ingress-canary-tcqk5"
Jul 14 17:33:17.457789 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.457769    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2s6j\" (UniqueName: \"kubernetes.io/projected/89dbd5c1-c9c4-4182-83e4-b1c086db1654-kube-api-access-k2s6j\") pod \"dns-default-f76mp\" (UID: \"89dbd5c1-c9c4-4182-83e4-b1c086db1654\") " pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:17.500656 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.500622    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:17.504552 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.504530    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:17.542563 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.542333    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tcqk5"
Jul 14 17:33:17.548783 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.548759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4186b2ec-a392-40d8-b06d-8be76fa35bbe-data-volume\") pod \"insights-runtime-extractor-s48hg\" (UID: \"4186b2ec-a392-40d8-b06d-8be76fa35bbe\") " pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.548886 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.548806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4186b2ec-a392-40d8-b06d-8be76fa35bbe-crio-socket\") pod \"insights-runtime-extractor-s48hg\" (UID: \"4186b2ec-a392-40d8-b06d-8be76fa35bbe\") " pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.548886 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.548831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4186b2ec-a392-40d8-b06d-8be76fa35bbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s48hg\" (UID: \"4186b2ec-a392-40d8-b06d-8be76fa35bbe\") " pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.549007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.548900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4xc4\" (UniqueName: \"kubernetes.io/projected/4186b2ec-a392-40d8-b06d-8be76fa35bbe-kube-api-access-v4xc4\") pod \"insights-runtime-extractor-s48hg\" (UID: \"4186b2ec-a392-40d8-b06d-8be76fa35bbe\") " pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.549007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.548934    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4186b2ec-a392-40d8-b06d-8be76fa35bbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s48hg\" (UID: \"4186b2ec-a392-40d8-b06d-8be76fa35bbe\") " pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.549007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.548934    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4186b2ec-a392-40d8-b06d-8be76fa35bbe-crio-socket\") pod \"insights-runtime-extractor-s48hg\" (UID: \"4186b2ec-a392-40d8-b06d-8be76fa35bbe\") " pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.549149 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.549117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4186b2ec-a392-40d8-b06d-8be76fa35bbe-data-volume\") pod \"insights-runtime-extractor-s48hg\" (UID: \"4186b2ec-a392-40d8-b06d-8be76fa35bbe\") " pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.549635 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.549610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4186b2ec-a392-40d8-b06d-8be76fa35bbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s48hg\" (UID: \"4186b2ec-a392-40d8-b06d-8be76fa35bbe\") " pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.551877 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.551823    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4186b2ec-a392-40d8-b06d-8be76fa35bbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s48hg\" (UID: \"4186b2ec-a392-40d8-b06d-8be76fa35bbe\") " pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.557724 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.557673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4xc4\" (UniqueName: \"kubernetes.io/projected/4186b2ec-a392-40d8-b06d-8be76fa35bbe-kube-api-access-v4xc4\") pod \"insights-runtime-extractor-s48hg\" (UID: \"4186b2ec-a392-40d8-b06d-8be76fa35bbe\") " pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.562041 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.561853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:17.602065 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.601979    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:17.624463 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.624024    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s48hg"
Jul 14 17:33:17.666281 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.666253    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f6b6674b9-4lr8w"]
Jul 14 17:33:17.671125 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:33:17.671076    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f3fcfd_03da_490d_aef8_06ef2c6875bb.slice/crio-3ad21e215270067a44b18cb501263b0e77454d48447caeba060c1a12abc39800 WatchSource:0}: Error finding container 3ad21e215270067a44b18cb501263b0e77454d48447caeba060c1a12abc39800: Status 404 returned error can't find the container with id 3ad21e215270067a44b18cb501263b0e77454d48447caeba060c1a12abc39800
Jul 14 17:33:17.698755 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.698657    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tcqk5"]
Jul 14 17:33:17.724369 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:33:17.724164    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2317c923_3e9d_4bee_b3fe_760df78d2176.slice/crio-2c85ad37563f48179871eb26c1c3a15ba6a90ed14116adb02d0e579167d0799d WatchSource:0}: Error finding container 2c85ad37563f48179871eb26c1c3a15ba6a90ed14116adb02d0e579167d0799d: Status 404 returned error can't find the container with id 2c85ad37563f48179871eb26c1c3a15ba6a90ed14116adb02d0e579167d0799d
Jul 14 17:33:17.737471 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.737444    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f76mp"]
Jul 14 17:33:17.741167 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:33:17.741133    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89dbd5c1_c9c4_4182_83e4_b1c086db1654.slice/crio-e8364b03048a5596501a27916babccf7b4b9b2b09d7bfc0077803d0aed81b358 WatchSource:0}: Error finding container e8364b03048a5596501a27916babccf7b4b9b2b09d7bfc0077803d0aed81b358: Status 404 returned error can't find the container with id e8364b03048a5596501a27916babccf7b4b9b2b09d7bfc0077803d0aed81b358
Jul 14 17:33:17.795786 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:17.795750    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s48hg"]
Jul 14 17:33:17.798779 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:33:17.798747    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4186b2ec_a392_40d8_b06d_8be76fa35bbe.slice/crio-ccaa04422f2ad826805ffac811e523645440f24208df926a590dc78a5434d297 WatchSource:0}: Error finding container ccaa04422f2ad826805ffac811e523645440f24208df926a590dc78a5434d297: Status 404 returned error can't find the container with id ccaa04422f2ad826805ffac811e523645440f24208df926a590dc78a5434d297
Jul 14 17:33:18.003765 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.003671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f76mp" event={"ID":"89dbd5c1-c9c4-4182-83e4-b1c086db1654","Type":"ContainerStarted","Data":"e8364b03048a5596501a27916babccf7b4b9b2b09d7bfc0077803d0aed81b358"}
Jul 14 17:33:18.004831 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.004802    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s48hg" event={"ID":"4186b2ec-a392-40d8-b06d-8be76fa35bbe","Type":"ContainerStarted","Data":"ccaa04422f2ad826805ffac811e523645440f24208df926a590dc78a5434d297"}
Jul 14 17:33:18.005773 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.005752    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tcqk5" event={"ID":"2317c923-3e9d-4bee-b3fe-760df78d2176","Type":"ContainerStarted","Data":"2c85ad37563f48179871eb26c1c3a15ba6a90ed14116adb02d0e579167d0799d"}
Jul 14 17:33:18.007337 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.007315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w" event={"ID":"a6f3fcfd-03da-490d-aef8-06ef2c6875bb","Type":"ContainerStarted","Data":"42698c0500fd0eb93eda6e349f03708e4bf3a9b2a9d68c5d7e734ae9cb55f589"}
Jul 14 17:33:18.007430 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.007348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w" event={"ID":"a6f3fcfd-03da-490d-aef8-06ef2c6875bb","Type":"ContainerStarted","Data":"3ad21e215270067a44b18cb501263b0e77454d48447caeba060c1a12abc39800"}
Jul 14 17:33:18.007497 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.007444    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:18.031996 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.031920    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w" podStartSLOduration=6.031904138 podStartE2EDuration="6.031904138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:33:18.030938449 +0000 UTC m=+54.747332140" watchObservedRunningTime="2026-07-14 17:33:18.031904138 +0000 UTC m=+54.748297829"
Jul 14 17:33:18.273605 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.273522    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:18.341371 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.341338    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:18.777700 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.777413    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:33:18.777936 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.777845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:33:18.778318 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.778300    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:33:18.784071 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.783350    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 17:33:18.784071 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.783573    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 17:33:18.784071 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.783639    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-79b6j\""
Jul 14 17:33:18.784071 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.783752    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 17:33:18.784071 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.783798    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 17:33:18.787364 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:18.787298    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q7745\""
Jul 14 17:33:20.065734 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:20.065662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:20.157072 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:20.157032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:20.735464 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:20.735432    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:20.842171 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:20.842139    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:21.353973 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:21.353939    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:21.465177 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:21.465138    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:22.661801 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.661770    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:22.702389 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.702362    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-5x6wl"]
Jul 14 17:33:22.715465 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.715434    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-5x6wl"]
Jul 14 17:33:22.715619 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.715555    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl"
Jul 14 17:33:22.718379 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.718354    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 14 17:33:22.718518 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.718386    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 14 17:33:22.718518 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.718394    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 14 17:33:22.719064 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.718915    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 14 17:33:22.719064 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.718930    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 14 17:33:22.719191 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.719076    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-kfcdt\""
Jul 14 17:33:22.739172 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.739144    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:22.791400 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.791366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxshn\" (UniqueName: \"kubernetes.io/projected/cffc77fc-29b6-4126-9b0b-3ed4fc964ee2-kube-api-access-wxshn\") pod \"prometheus-operator-64785b6675-5x6wl\" (UID: \"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl"
Jul 14 17:33:22.791563 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.791425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cffc77fc-29b6-4126-9b0b-3ed4fc964ee2-metrics-client-ca\") pod \"prometheus-operator-64785b6675-5x6wl\" (UID: \"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl"
Jul 14 17:33:22.791563 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.791449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cffc77fc-29b6-4126-9b0b-3ed4fc964ee2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-5x6wl\" (UID: \"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl"
Jul 14 17:33:22.791563 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.791477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cffc77fc-29b6-4126-9b0b-3ed4fc964ee2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-5x6wl\" (UID: \"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl"
Jul 14 17:33:22.892361 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.892326    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxshn\" (UniqueName: \"kubernetes.io/projected/cffc77fc-29b6-4126-9b0b-3ed4fc964ee2-kube-api-access-wxshn\") pod \"prometheus-operator-64785b6675-5x6wl\" (UID: \"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl"
Jul 14 17:33:22.892530 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.892385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cffc77fc-29b6-4126-9b0b-3ed4fc964ee2-metrics-client-ca\") pod \"prometheus-operator-64785b6675-5x6wl\" (UID: \"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl"
Jul 14 17:33:22.892530 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.892413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cffc77fc-29b6-4126-9b0b-3ed4fc964ee2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-5x6wl\" (UID: \"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl"
Jul 14 17:33:22.892530 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.892430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cffc77fc-29b6-4126-9b0b-3ed4fc964ee2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-5x6wl\" (UID: \"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl"
Jul 14 17:33:22.892708 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:22.892551    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 14 17:33:22.892708 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:22.892638    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cffc77fc-29b6-4126-9b0b-3ed4fc964ee2-prometheus-operator-tls podName:cffc77fc-29b6-4126-9b0b-3ed4fc964ee2 nodeName:}" failed. No retries permitted until 2026-07-14 17:33:23.392615212 +0000 UTC m=+60.109008889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/cffc77fc-29b6-4126-9b0b-3ed4fc964ee2-prometheus-operator-tls") pod "prometheus-operator-64785b6675-5x6wl" (UID: "cffc77fc-29b6-4126-9b0b-3ed4fc964ee2") : secret "prometheus-operator-tls" not found
Jul 14 17:33:22.893111 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.893085    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cffc77fc-29b6-4126-9b0b-3ed4fc964ee2-metrics-client-ca\") pod \"prometheus-operator-64785b6675-5x6wl\" (UID: \"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl"
Jul 14 17:33:22.896842 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.896776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cffc77fc-29b6-4126-9b0b-3ed4fc964ee2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-5x6wl\" (UID: \"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl"
Jul 14 17:33:22.909966 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:22.909943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxshn\" (UniqueName: \"kubernetes.io/projected/cffc77fc-29b6-4126-9b0b-3ed4fc964ee2-kube-api-access-wxshn\") pod \"prometheus-operator-64785b6675-5x6wl\" (UID: \"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl"
Jul 14 17:33:23.371192 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:23.371165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:23.397234 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:23.397196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cffc77fc-29b6-4126-9b0b-3ed4fc964ee2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-5x6wl\" (UID: \"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl"
Jul 14 17:33:23.399489 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:23.399469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cffc77fc-29b6-4126-9b0b-3ed4fc964ee2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-5x6wl\" (UID: \"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl"
Jul 14 17:33:23.629553 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:23.629528    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-kfcdt\""
Jul 14 17:33:23.637713 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:23.637690    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl"
Jul 14 17:33:23.776372 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:23.776333    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-5x6wl"]
Jul 14 17:33:23.780165 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:33:23.780132    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcffc77fc_29b6_4126_9b0b_3ed4fc964ee2.slice/crio-5d9861e17836241f2aa116a933c47bdef43b76c0665b8214e7196241b9584626 WatchSource:0}: Error finding container 5d9861e17836241f2aa116a933c47bdef43b76c0665b8214e7196241b9584626: Status 404 returned error can't find the container with id 5d9861e17836241f2aa116a933c47bdef43b76c0665b8214e7196241b9584626
Jul 14 17:33:24.028499 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:24.028412    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f76mp" event={"ID":"89dbd5c1-c9c4-4182-83e4-b1c086db1654","Type":"ContainerStarted","Data":"bb21bf67754992bcb53ea9a4911345b6721cbb9a73568f4bf8da010bb333c190"}
Jul 14 17:33:24.028499 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:24.028467    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f76mp" event={"ID":"89dbd5c1-c9c4-4182-83e4-b1c086db1654","Type":"ContainerStarted","Data":"4419c49002e5bff10d511468e9eac6ad639553c0d928c8029eb10fab9530b24a"}
Jul 14 17:33:24.028499 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:24.028497    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:24.029822 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:24.029794    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s48hg" event={"ID":"4186b2ec-a392-40d8-b06d-8be76fa35bbe","Type":"ContainerStarted","Data":"6890946b77fd4adcc16d273d5e89da605874ed3730227447e0bfba351896a3e9"}
Jul 14 17:33:24.031138 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:24.031113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tcqk5" event={"ID":"2317c923-3e9d-4bee-b3fe-760df78d2176","Type":"ContainerStarted","Data":"10db5622b75c4aa817f3fc97226e20197b803234203be28b0da176b6e01f349b"}
Jul 14 17:33:24.032264 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:24.032241    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl" event={"ID":"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2","Type":"ContainerStarted","Data":"5d9861e17836241f2aa116a933c47bdef43b76c0665b8214e7196241b9584626"}
Jul 14 17:33:24.051198 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:24.049487    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-f76mp" podStartSLOduration=1.7681998509999999 podStartE2EDuration="7.049465103s" totalImagesPullingTime="5.281265252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:33:24.047872348 +0000 UTC m=+60.764266033" watchObservedRunningTime="2026-07-14 17:33:24.049465103 +0000 UTC m=+60.765858792"
Jul 14 17:33:24.066571 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:24.066526    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tcqk5" podStartSLOduration=1.766886328 podStartE2EDuration="7.066511645s" totalImagesPullingTime="5.299625317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:33:24.06570273 +0000 UTC m=+60.782096441" watchObservedRunningTime="2026-07-14 17:33:24.066511645 +0000 UTC m=+60.782905354"
Jul 14 17:33:25.036795 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:25.036756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s48hg" event={"ID":"4186b2ec-a392-40d8-b06d-8be76fa35bbe","Type":"ContainerStarted","Data":"89a051fdead5d391d7f0fc00c6a3fbc72573dd5892a63ed9c55e1996e8e8b006"}
Jul 14 17:33:25.262661 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:25.262623    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:25.327573 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:25.327487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:26.042080 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:26.042037    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl" event={"ID":"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2","Type":"ContainerStarted","Data":"5a26a80dac66220520b64d3c06c48da705acd4e2c94ec6c9774f6aa9c9d0e278"}
Jul 14 17:33:26.042472 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:26.042092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl" event={"ID":"cffc77fc-29b6-4126-9b0b-3ed4fc964ee2","Type":"ContainerStarted","Data":"1ec8648a31653f7be72e4d9a2b1fba7b4d1830006e329df1200c8e722e00aa31"}
Jul 14 17:33:26.144274 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:26.144177    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:27.065098 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:27.065044    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-5x6wl" podStartSLOduration=3.041192696 podStartE2EDuration="5.065024858s" totalImagesPullingTime="2.023832162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:33:27.064534941 +0000 UTC m=+63.780928634" watchObservedRunningTime="2026-07-14 17:33:27.065024858 +0000 UTC m=+63.781418547"
Jul 14 17:33:27.638568 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:27.638521    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:27.717177 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:27.717137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:28.437631 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.437592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xpb\" (UniqueName: \"kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb\") pod \"network-check-target-hv7kv\" (UID: \"ab91d593-f6ad-4a22-9d7f-e90a0dcc9787\") " pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:33:28.437631 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.437636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:33:28.438051 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.437656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs\") pod \"network-metrics-daemon-x8dr4\" (UID: \"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d\") " pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:33:28.440662 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.440643    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 17:33:28.441289 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.441274    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 17:33:28.441353 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.441305    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 17:33:28.450978 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.450949    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7-original-pull-secret\") pod \"global-pull-secret-syncer-fk7h5\" (UID: \"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7\") " pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:33:28.450978 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.450963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d-metrics-certs\") pod \"network-metrics-daemon-x8dr4\" (UID: \"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d\") " pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:33:28.451458 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.451442    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 17:33:28.461585 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.461565    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4xpb\" (UniqueName: \"kubernetes.io/projected/ab91d593-f6ad-4a22-9d7f-e90a0dcc9787-kube-api-access-b4xpb\") pod \"network-check-target-hv7kv\" (UID: \"ab91d593-f6ad-4a22-9d7f-e90a0dcc9787\") " pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:33:28.703573 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.703493    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q7745\""
Jul 14 17:33:28.710863 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.710843    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:33:28.714337 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.714320    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-79b6j\""
Jul 14 17:33:28.721497 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.721477    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fk7h5"
Jul 14 17:33:28.722548 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.722525    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8dr4"
Jul 14 17:33:28.816670 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.816515    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:28.877181 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.876775    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hv7kv"]
Jul 14 17:33:28.881447 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:33:28.881411    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab91d593_f6ad_4a22_9d7f_e90a0dcc9787.slice/crio-31e136de6e9422f0dcceeba97a074d7e3bf08a9698ee40df422af165c8c8f6d6 WatchSource:0}: Error finding container 31e136de6e9422f0dcceeba97a074d7e3bf08a9698ee40df422af165c8c8f6d6: Status 404 returned error can't find the container with id 31e136de6e9422f0dcceeba97a074d7e3bf08a9698ee40df422af165c8c8f6d6
Jul 14 17:33:28.903814 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.903787    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fk7h5"]
Jul 14 17:33:28.906610 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:33:28.906579    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb86cfa4_c36d_43f2_b8ac_3ceb635b59f7.slice/crio-4531ddfbed12a3e41b7348d04595e572da9a40fc6f86ce9d60d62d307904ebf9 WatchSource:0}: Error finding container 4531ddfbed12a3e41b7348d04595e572da9a40fc6f86ce9d60d62d307904ebf9: Status 404 returned error can't find the container with id 4531ddfbed12a3e41b7348d04595e572da9a40fc6f86ce9d60d62d307904ebf9
Jul 14 17:33:28.931269 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:28.931248    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x8dr4"]
Jul 14 17:33:28.932885 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:33:28.932863    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535c6d0a_9f3a_4b23_9cc2_ba87f3f16d9d.slice/crio-b175e76aa24216955936f93b3cf7e343afb1b5ecca4562a22628615d92341d94 WatchSource:0}: Error finding container b175e76aa24216955936f93b3cf7e343afb1b5ecca4562a22628615d92341d94: Status 404 returned error can't find the container with id b175e76aa24216955936f93b3cf7e343afb1b5ecca4562a22628615d92341d94
Jul 14 17:33:29.022322 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.022232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:29.053565 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.053524    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s48hg" event={"ID":"4186b2ec-a392-40d8-b06d-8be76fa35bbe","Type":"ContainerStarted","Data":"a2a9abefa35c9881758954e8b325b16943e9b7095919916bb4ef5f8bb66e5676"}
Jul 14 17:33:29.054862 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.054824    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hv7kv" event={"ID":"ab91d593-f6ad-4a22-9d7f-e90a0dcc9787","Type":"ContainerStarted","Data":"31e136de6e9422f0dcceeba97a074d7e3bf08a9698ee40df422af165c8c8f6d6"}
Jul 14 17:33:29.055873 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.055848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x8dr4" event={"ID":"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d","Type":"ContainerStarted","Data":"b175e76aa24216955936f93b3cf7e343afb1b5ecca4562a22628615d92341d94"}
Jul 14 17:33:29.056953 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.056936    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fk7h5" event={"ID":"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7","Type":"ContainerStarted","Data":"4531ddfbed12a3e41b7348d04595e572da9a40fc6f86ce9d60d62d307904ebf9"}
Jul 14 17:33:29.078646 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.078596    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s48hg" podStartSLOduration=1.6307728670000001 podStartE2EDuration="12.078582405s" totalImagesPullingTime="10.447809538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:33:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:33:29.077500022 +0000 UTC m=+65.793893713" watchObservedRunningTime="2026-07-14 17:33:29.078582405 +0000 UTC m=+65.794976098"
Jul 14 17:33:29.126628 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.126596    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:29.135130 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.135103    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"]
Jul 14 17:33:29.163753 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.163713    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"]
Jul 14 17:33:29.163866 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.163834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"
Jul 14 17:33:29.173626 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.173606    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 14 17:33:29.173626 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.173615    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 14 17:33:29.173896 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.173880    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mpw79\""
Jul 14 17:33:29.216582 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.216557    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"]
Jul 14 17:33:29.241659 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.241626    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5cr5d"]
Jul 14 17:33:29.241781 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.241755    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.242518 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.242490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ee90214-e9d3-4400-8170-7a6fd3a39652-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-nd8j5\" (UID: \"9ee90214-e9d3-4400-8170-7a6fd3a39652\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"
Jul 14 17:33:29.242637 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.242532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ee90214-e9d3-4400-8170-7a6fd3a39652-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-nd8j5\" (UID: \"9ee90214-e9d3-4400-8170-7a6fd3a39652\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"
Jul 14 17:33:29.242637 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.242572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk4hp\" (UniqueName: \"kubernetes.io/projected/9ee90214-e9d3-4400-8170-7a6fd3a39652-kube-api-access-pk4hp\") pod \"openshift-state-metrics-84bd68b896-nd8j5\" (UID: \"9ee90214-e9d3-4400-8170-7a6fd3a39652\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"
Jul 14 17:33:29.242637 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.242623    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ee90214-e9d3-4400-8170-7a6fd3a39652-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-nd8j5\" (UID: \"9ee90214-e9d3-4400-8170-7a6fd3a39652\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"
Jul 14 17:33:29.244709 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.244687    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:29.245094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.245081    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 14 17:33:29.245641 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.245605    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-v68kz\""
Jul 14 17:33:29.245744 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.245620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 14 17:33:29.245744 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.245665    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 14 17:33:29.264760 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.264741    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"]
Jul 14 17:33:29.264849 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.264839    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.268898 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.268880    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 14 17:33:29.269243 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.269225    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 14 17:33:29.269613 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.269597    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bq24s\""
Jul 14 17:33:29.271344 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.271328    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 14 17:33:29.342931 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.342852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ee90214-e9d3-4400-8170-7a6fd3a39652-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-nd8j5\" (UID: \"9ee90214-e9d3-4400-8170-7a6fd3a39652\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"
Jul 14 17:33:29.342931 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.342884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.342931 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.342902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmfzb\" (UniqueName: \"kubernetes.io/projected/d483aff6-f6b3-4ab6-836e-3817bfa065a6-kube-api-access-jmfzb\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.342931 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.342924    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d483aff6-f6b3-4ab6-836e-3817bfa065a6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.343224 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.342944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhs46\" (UniqueName: \"kubernetes.io/projected/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-kube-api-access-nhs46\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.343224 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.342982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ee90214-e9d3-4400-8170-7a6fd3a39652-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-nd8j5\" (UID: \"9ee90214-e9d3-4400-8170-7a6fd3a39652\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"
Jul 14 17:33:29.343224 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.343009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-node-exporter-wtmp\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.343224 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.343047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-node-exporter-textfile\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.343224 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.343076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d483aff6-f6b3-4ab6-836e-3817bfa065a6-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.343224 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.343109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-node-exporter-accelerators-collector-config\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.343224 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:29.343134    2578 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 14 17:33:29.343224 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.343151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d483aff6-f6b3-4ab6-836e-3817bfa065a6-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.343224 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:29.343192    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ee90214-e9d3-4400-8170-7a6fd3a39652-openshift-state-metrics-tls podName:9ee90214-e9d3-4400-8170-7a6fd3a39652 nodeName:}" failed. No retries permitted until 2026-07-14 17:33:29.843171809 +0000 UTC m=+66.559565479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9ee90214-e9d3-4400-8170-7a6fd3a39652-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-nd8j5" (UID: "9ee90214-e9d3-4400-8170-7a6fd3a39652") : secret "openshift-state-metrics-tls" not found
Jul 14 17:33:29.343564 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.343269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-root\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.343564 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.343309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk4hp\" (UniqueName: \"kubernetes.io/projected/9ee90214-e9d3-4400-8170-7a6fd3a39652-kube-api-access-pk4hp\") pod \"openshift-state-metrics-84bd68b896-nd8j5\" (UID: \"9ee90214-e9d3-4400-8170-7a6fd3a39652\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"
Jul 14 17:33:29.343564 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.343341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d483aff6-f6b3-4ab6-836e-3817bfa065a6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.343564 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.343368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-node-exporter-tls\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.343564 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.343423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ee90214-e9d3-4400-8170-7a6fd3a39652-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-nd8j5\" (UID: \"9ee90214-e9d3-4400-8170-7a6fd3a39652\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"
Jul 14 17:33:29.343564 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.343452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-metrics-client-ca\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.343564 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.343477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d483aff6-f6b3-4ab6-836e-3817bfa065a6-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.343564 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.343507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-sys\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.343824 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.343680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ee90214-e9d3-4400-8170-7a6fd3a39652-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-nd8j5\" (UID: \"9ee90214-e9d3-4400-8170-7a6fd3a39652\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"
Jul 14 17:33:29.343824 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.343737    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:29.345848 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.345831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ee90214-e9d3-4400-8170-7a6fd3a39652-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-nd8j5\" (UID: \"9ee90214-e9d3-4400-8170-7a6fd3a39652\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"
Jul 14 17:33:29.353114 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.353095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk4hp\" (UniqueName: \"kubernetes.io/projected/9ee90214-e9d3-4400-8170-7a6fd3a39652-kube-api-access-pk4hp\") pod \"openshift-state-metrics-84bd68b896-nd8j5\" (UID: \"9ee90214-e9d3-4400-8170-7a6fd3a39652\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"
Jul 14 17:33:29.434964 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.434934    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:29.444287 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-metrics-client-ca\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.444677 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d483aff6-f6b3-4ab6-836e-3817bfa065a6-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.444677 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-sys\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.444677 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.444677 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmfzb\" (UniqueName: \"kubernetes.io/projected/d483aff6-f6b3-4ab6-836e-3817bfa065a6-kube-api-access-jmfzb\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.444677 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d483aff6-f6b3-4ab6-836e-3817bfa065a6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.444677 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhs46\" (UniqueName: \"kubernetes.io/projected/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-kube-api-access-nhs46\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.444677 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444503    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-node-exporter-wtmp\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.444677 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-node-exporter-textfile\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.444677 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d483aff6-f6b3-4ab6-836e-3817bfa065a6-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.444677 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-node-exporter-accelerators-collector-config\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.444677 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d483aff6-f6b3-4ab6-836e-3817bfa065a6-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.444677 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-root\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.444677 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d483aff6-f6b3-4ab6-836e-3817bfa065a6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.445280 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-node-exporter-tls\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.445280 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.444925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-metrics-client-ca\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.445280 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.445243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-node-exporter-textfile\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.445429 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.445295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-node-exporter-accelerators-collector-config\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.445573 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.445548    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-node-exporter-wtmp\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.445698 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.445572    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d483aff6-f6b3-4ab6-836e-3817bfa065a6-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.445698 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.445585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d483aff6-f6b3-4ab6-836e-3817bfa065a6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.445698 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.445606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-root\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.445698 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.445637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-sys\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.446132 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.446066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d483aff6-f6b3-4ab6-836e-3817bfa065a6-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.447494 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.447473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.448224 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.448185    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d483aff6-f6b3-4ab6-836e-3817bfa065a6-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.448352 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.448336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-node-exporter-tls\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.448556 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.448535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d483aff6-f6b3-4ab6-836e-3817bfa065a6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.453408 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.453364    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhs46\" (UniqueName: \"kubernetes.io/projected/5f3e5c41-2310-4cb6-b57b-2f3407c09c69-kube-api-access-nhs46\") pod \"node-exporter-5cr5d\" (UID: \"5f3e5c41-2310-4cb6-b57b-2f3407c09c69\") " pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.454495 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.454467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmfzb\" (UniqueName: \"kubernetes.io/projected/d483aff6-f6b3-4ab6-836e-3817bfa065a6-kube-api-access-jmfzb\") pod \"kube-state-metrics-785778ff74-gsw6x\" (UID: \"d483aff6-f6b3-4ab6-836e-3817bfa065a6\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.551304 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.551267    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"
Jul 14 17:33:29.572447 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.572415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5cr5d"
Jul 14 17:33:29.588861 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.588831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:29.626767 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:33:29.626731    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3e5c41_2310_4cb6_b57b_2f3407c09c69.slice/crio-fc856339a6f15e3e37513e5e5e36f1d532a3b04a6db8410e06fe308621f1ed4c WatchSource:0}: Error finding container fc856339a6f15e3e37513e5e5e36f1d532a3b04a6db8410e06fe308621f1ed4c: Status 404 returned error can't find the container with id fc856339a6f15e3e37513e5e5e36f1d532a3b04a6db8410e06fe308621f1ed4c
Jul 14 17:33:29.717171 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.717125    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:29.743465 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.743434    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-gsw6x"]
Jul 14 17:33:29.747578 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:33:29.747554    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd483aff6_f6b3_4ab6_836e_3817bfa065a6.slice/crio-503cefd9339a9a9ecf85f4d6eaac0f1fecd8d1f7e954a85a1453e755c2caed99 WatchSource:0}: Error finding container 503cefd9339a9a9ecf85f4d6eaac0f1fecd8d1f7e954a85a1453e755c2caed99: Status 404 returned error can't find the container with id 503cefd9339a9a9ecf85f4d6eaac0f1fecd8d1f7e954a85a1453e755c2caed99
Jul 14 17:33:29.802219 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.802172    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:29.848328 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.848259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ee90214-e9d3-4400-8170-7a6fd3a39652-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-nd8j5\" (UID: \"9ee90214-e9d3-4400-8170-7a6fd3a39652\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"
Jul 14 17:33:29.851167 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.851128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ee90214-e9d3-4400-8170-7a6fd3a39652-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-nd8j5\" (UID: \"9ee90214-e9d3-4400-8170-7a6fd3a39652\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"
Jul 14 17:33:29.885721 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:29.885686    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:30.041468 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:30.041435    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:30.068516 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:30.068475    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5cr5d" event={"ID":"5f3e5c41-2310-4cb6-b57b-2f3407c09c69","Type":"ContainerStarted","Data":"fc856339a6f15e3e37513e5e5e36f1d532a3b04a6db8410e06fe308621f1ed4c"}
Jul 14 17:33:30.070121 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:30.070087    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x" event={"ID":"d483aff6-f6b3-4ab6-836e-3817bfa065a6","Type":"ContainerStarted","Data":"503cefd9339a9a9ecf85f4d6eaac0f1fecd8d1f7e954a85a1453e755c2caed99"}
Jul 14 17:33:30.072677 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:30.072646    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"
Jul 14 17:33:30.152462 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:30.152126    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:30.252482 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:30.252446    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5"]
Jul 14 17:33:30.255655 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:30.255616    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:30.259727 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:33:30.259701    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee90214_e9d3_4400_8170_7a6fd3a39652.slice/crio-048be6192dfd6924e81438ae13cb94fe85f84756732ff26b08f6306d2ea73e7b WatchSource:0}: Error finding container 048be6192dfd6924e81438ae13cb94fe85f84756732ff26b08f6306d2ea73e7b: Status 404 returned error can't find the container with id 048be6192dfd6924e81438ae13cb94fe85f84756732ff26b08f6306d2ea73e7b
Jul 14 17:33:30.819844 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:30.819748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:30.990696 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:30.989689    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9hr5"
Jul 14 17:33:30.990696 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:30.990334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:31.077637 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:31.077552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5" event={"ID":"9ee90214-e9d3-4400-8170-7a6fd3a39652","Type":"ContainerStarted","Data":"7ee57ec4b8be352ff687ec4c16a037dee1ef8ae9992ddbb8441efff46897c3e8"}
Jul 14 17:33:31.077637 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:31.077620    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5" event={"ID":"9ee90214-e9d3-4400-8170-7a6fd3a39652","Type":"ContainerStarted","Data":"6160ce61e5273282bd5052e5e13e419d160daf19cdd2fdc988d49d5d6fa57162"}
Jul 14 17:33:31.077853 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:31.077644    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5" event={"ID":"9ee90214-e9d3-4400-8170-7a6fd3a39652","Type":"ContainerStarted","Data":"048be6192dfd6924e81438ae13cb94fe85f84756732ff26b08f6306d2ea73e7b"}
Jul 14 17:33:31.140751 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:31.140678    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:31.254579 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:31.254540    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:31.383074 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:31.382989    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:32.097668 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:32.097618    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:32.198483 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:32.198448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:32.297894 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:32.297861    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:33.832897 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:33.832864    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:33.840062 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:33.840024    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt"]
Jul 14 17:33:33.842298 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:33.842276    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt"
Jul 14 17:33:33.844636 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:33.844615    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rdbrp\""
Jul 14 17:33:33.844784 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:33.844763    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 14 17:33:33.855496 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:33.855467    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt"]
Jul 14 17:33:33.884763 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:33.884732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c0a12187-5a00-4ab2-a043-37511faad495-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-hc5dt\" (UID: \"c0a12187-5a00-4ab2-a043-37511faad495\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt"
Jul 14 17:33:33.884935 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:33.884771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l7tb\" (UniqueName: \"kubernetes.io/projected/c0a12187-5a00-4ab2-a043-37511faad495-kube-api-access-6l7tb\") pod \"monitoring-plugin-5f68df7f58-hc5dt\" (UID: \"c0a12187-5a00-4ab2-a043-37511faad495\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt"
Jul 14 17:33:33.934901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:33.934842    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:33.986025 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:33.985985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c0a12187-5a00-4ab2-a043-37511faad495-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-hc5dt\" (UID: \"c0a12187-5a00-4ab2-a043-37511faad495\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt"
Jul 14 17:33:33.986221 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:33.986035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l7tb\" (UniqueName: \"kubernetes.io/projected/c0a12187-5a00-4ab2-a043-37511faad495-kube-api-access-6l7tb\") pod \"monitoring-plugin-5f68df7f58-hc5dt\" (UID: \"c0a12187-5a00-4ab2-a043-37511faad495\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt"
Jul 14 17:33:33.986221 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:33.986147    2578 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 14 17:33:33.986334 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:33:33.986259    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a12187-5a00-4ab2-a043-37511faad495-monitoring-plugin-cert podName:c0a12187-5a00-4ab2-a043-37511faad495 nodeName:}" failed. No retries permitted until 2026-07-14 17:33:34.486237775 +0000 UTC m=+71.202631445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/c0a12187-5a00-4ab2-a043-37511faad495-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-hc5dt" (UID: "c0a12187-5a00-4ab2-a043-37511faad495") : secret "monitoring-plugin-cert" not found
Jul 14 17:33:33.995844 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:33.995812    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l7tb\" (UniqueName: \"kubernetes.io/projected/c0a12187-5a00-4ab2-a043-37511faad495-kube-api-access-6l7tb\") pod \"monitoring-plugin-5f68df7f58-hc5dt\" (UID: \"c0a12187-5a00-4ab2-a043-37511faad495\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt"
Jul 14 17:33:34.039817 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:34.039778    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-f76mp"
Jul 14 17:33:34.096431 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:34.096345    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:34.490109 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:34.490073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c0a12187-5a00-4ab2-a043-37511faad495-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-hc5dt\" (UID: \"c0a12187-5a00-4ab2-a043-37511faad495\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt"
Jul 14 17:33:34.493054 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:34.493029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c0a12187-5a00-4ab2-a043-37511faad495-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-hc5dt\" (UID: \"c0a12187-5a00-4ab2-a043-37511faad495\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt"
Jul 14 17:33:34.752955 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:34.752927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt"
Jul 14 17:33:35.136507 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:35.136474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x" event={"ID":"d483aff6-f6b3-4ab6-836e-3817bfa065a6","Type":"ContainerStarted","Data":"72c1a749066b5cc325c216e24d2c43b6a4d48becf02374d2986715b46dee8d59"}
Jul 14 17:33:35.188864 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:35.188800    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt"]
Jul 14 17:33:35.532071 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:35.531969    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:36.142237 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.142185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hv7kv" event={"ID":"ab91d593-f6ad-4a22-9d7f-e90a0dcc9787","Type":"ContainerStarted","Data":"3686f480bf65fc6da22de613a63f00b73d3b56b774a8b6c3664ae585e3a3ae79"}
Jul 14 17:33:36.142647 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.142294    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:33:36.144016 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.143991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x8dr4" event={"ID":"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d","Type":"ContainerStarted","Data":"e03128300f5fa0d04a474cb2a8a528c8feaca3f2f8aca4f79a16e5921e1d68bf"}
Jul 14 17:33:36.144129 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.144040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x8dr4" event={"ID":"535c6d0a-9f3a-4b23-9cc2-ba87f3f16d9d","Type":"ContainerStarted","Data":"fd9cc4ba6dcb4c8d81a66ca159b3487a6796310a98697cc5b8de9969b6ca66f1"}
Jul 14 17:33:36.145252 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.145226    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fk7h5" event={"ID":"fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7","Type":"ContainerStarted","Data":"b7a52208ae8b3921e8d9513768c0a58224fc3b1713ea45f0ccac9a4a913025a4"}
Jul 14 17:33:36.147298 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.147275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5" event={"ID":"9ee90214-e9d3-4400-8170-7a6fd3a39652","Type":"ContainerStarted","Data":"52f6297e44361abd81f56c9add7b542c7722c5348afae097f19e1be5caaad9b4"}
Jul 14 17:33:36.148331 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.148313    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt" event={"ID":"c0a12187-5a00-4ab2-a043-37511faad495","Type":"ContainerStarted","Data":"81d0c715763ae09c43f3bc4aa5775e369fccb1af2d935d4480b57cf0a97ead74"}
Jul 14 17:33:36.149567 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.149547    2578 generic.go:356] "Generic (PLEG): container finished" podID="5f3e5c41-2310-4cb6-b57b-2f3407c09c69" containerID="62315cd51c164d3bc3fcf17bccff5113d0a883faaeb4414d49a7ceffd24170e3" exitCode=0
Jul 14 17:33:36.149646 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.149612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5cr5d" event={"ID":"5f3e5c41-2310-4cb6-b57b-2f3407c09c69","Type":"ContainerDied","Data":"62315cd51c164d3bc3fcf17bccff5113d0a883faaeb4414d49a7ceffd24170e3"}
Jul 14 17:33:36.151683 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.151663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x" event={"ID":"d483aff6-f6b3-4ab6-836e-3817bfa065a6","Type":"ContainerStarted","Data":"33812ae388d1da56c73ee7156b072853da2af3badf719c371c64a5403461a27b"}
Jul 14 17:33:36.151763 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.151700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x" event={"ID":"d483aff6-f6b3-4ab6-836e-3817bfa065a6","Type":"ContainerStarted","Data":"cb6cfb5ed3a0a767524bb1184d20143281f8ad06be3c1564ca5df60429264298"}
Jul 14 17:33:36.171382 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.171353    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:36.176522 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.176483    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hv7kv" podStartSLOduration=66.086493485 podStartE2EDuration="1m12.176469518s" totalImagesPullingTime="6.089976033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:33:36.174603798 +0000 UTC m=+72.890997489" watchObservedRunningTime="2026-07-14 17:33:36.176469518 +0000 UTC m=+72.892863203"
Jul 14 17:33:36.231120 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.231066    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-nd8j5" podStartSLOduration=2.831199007 podStartE2EDuration="7.231042441s" totalImagesPullingTime="4.399843434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:33:36.197104616 +0000 UTC m=+72.913498308" watchObservedRunningTime="2026-07-14 17:33:36.231042441 +0000 UTC m=+72.947436145"
Jul 14 17:33:36.260120 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.259517    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x8dr4" podStartSLOduration=66.250179626 podStartE2EDuration="1m12.259497075s" totalImagesPullingTime="6.009317449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:33:36.257562925 +0000 UTC m=+72.973956808" watchObservedRunningTime="2026-07-14 17:33:36.259497075 +0000 UTC m=+72.975890768"
Jul 14 17:33:36.261016 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.260976    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:36.276568 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.276490    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fk7h5" podStartSLOduration=66.197442886 podStartE2EDuration="1m12.276469196s" totalImagesPullingTime="6.07902631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:33:36.27512438 +0000 UTC m=+72.991518068" watchObservedRunningTime="2026-07-14 17:33:36.276469196 +0000 UTC m=+72.992862889"
Jul 14 17:33:36.297199 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.297129    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-gsw6x" podStartSLOduration=2.07225409 podStartE2EDuration="7.29711626s" totalImagesPullingTime="5.22486217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:33:36.295861848 +0000 UTC m=+73.012255539" watchObservedRunningTime="2026-07-14 17:33:36.29711626 +0000 UTC m=+73.013509954"
Jul 14 17:33:36.762827 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.762786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:36.854616 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:36.854586    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:37.157695 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:37.157601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5cr5d" event={"ID":"5f3e5c41-2310-4cb6-b57b-2f3407c09c69","Type":"ContainerStarted","Data":"780a0d33f67652bf3e2c89c30ccd49103cf89b3466f9935de548bec7315eeb7e"}
Jul 14 17:33:37.157695 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:37.157658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5cr5d" event={"ID":"5f3e5c41-2310-4cb6-b57b-2f3407c09c69","Type":"ContainerStarted","Data":"55f57c8b76d050a5ca9589e521586bb257fbcd996515b36480e52bc0a1f3c6a8"}
Jul 14 17:33:37.159300 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:37.159267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt" event={"ID":"c0a12187-5a00-4ab2-a043-37511faad495","Type":"ContainerStarted","Data":"009c5d41559d36e642321f86dd7df9109d3f2f2b83d19620369aa17e8573ca6e"}
Jul 14 17:33:37.160112 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:37.160090    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt"
Jul 14 17:33:37.165361 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:37.165343    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt"
Jul 14 17:33:37.183519 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:37.183468    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5cr5d" podStartSLOduration=2.676897352 podStartE2EDuration="8.183452274s" totalImagesPullingTime="5.344667117s" totalInitContainerRuntime="161.887805ms" isStatefulPod=true podCreationTimestamp="2026-07-14 17:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:33:37.181995349 +0000 UTC m=+73.898389072" watchObservedRunningTime="2026-07-14 17:33:37.183452274 +0000 UTC m=+73.899845973"
Jul 14 17:33:37.200001 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:37.199940    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hc5dt" podStartSLOduration=2.5781202260000002 podStartE2EDuration="4.199920819s" totalImagesPullingTime="1.621800593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:33:37.198440662 +0000 UTC m=+73.914834353" watchObservedRunningTime="2026-07-14 17:33:37.199920819 +0000 UTC m=+73.916314513"
Jul 14 17:33:39.016698 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:39.016673    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f6b6674b9-4lr8w"
Jul 14 17:33:39.026199 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:39.026174    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:39.158975 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:39.158936    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:39.285716 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:39.285629    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:39.382977 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:39.382947    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:39.541966 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:39.541885    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:39.641849 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:39.641819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:39.705262 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:39.705198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:40.452091 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:40.452057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:40.551806 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:40.551768    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:42.173745 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:42.173710    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:42.250060 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:42.250012    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:43.287130 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:43.287093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:43.387371 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:43.387329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:47.357638 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:47.357518    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:47.478608 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:47.478572    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:49.567189 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:49.567157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:49.664118 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:49.664086    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:51.077728 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:51.077696    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:51.177958 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:51.177928    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:51.248583 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:51.248550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:33:52.228504 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:33:52.228452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:34:02.400870 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:34:02.400841    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:34:02.495263 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:34:02.495226    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:34:07.161713 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:34:07.161678    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hv7kv"
Jul 14 17:34:17.378604 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:34:17.378564    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:34:17.459404 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:34:17.459375    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:34:31.588912 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:34:31.588874    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:34:31.673876 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:34:31.673835    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:34:45.318299 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:34:45.318265    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:34:45.421954 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:34:45.421927    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:34:57.339936 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:34:57.339901    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:34:57.440725 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:34:57.440693    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:35:02.384382 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:35:02.384347    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:35:03.206353 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:35:03.206321    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:35:10.260358 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:35:10.260325    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:35:10.368217 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:35:10.368183    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:35:10.473243 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:35:10.473187    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:35:11.002388 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:35:11.002348    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:35:11.077738 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:35:11.077706    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:35:21.559478 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:35:21.559402    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:35:21.679717 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:35:21.679680    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:35:21.787015 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:35:21.786982    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:35:37.718086 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:35:37.718040    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:35:37.849046 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:35:37.849010    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:35:46.795801 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:35:46.795768    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:02.593671 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:02.593638    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:02.706531 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:02.706499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:03.924603 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:03.924563    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"]
Jul 14 17:36:03.927023 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:03.927001    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"
Jul 14 17:36:03.929783 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:03.929760    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 14 17:36:03.930578 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:03.930558    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zdg2s\""
Jul 14 17:36:03.930578 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:03.930566    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 14 17:36:03.947493 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:03.947470    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"]
Jul 14 17:36:03.985538 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:03.985488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8dbc896-b34b-4e32-a7ad-5155d9d10536-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk\" (UID: \"b8dbc896-b34b-4e32-a7ad-5155d9d10536\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"
Jul 14 17:36:03.985538 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:03.985541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hhjk\" (UniqueName: \"kubernetes.io/projected/b8dbc896-b34b-4e32-a7ad-5155d9d10536-kube-api-access-7hhjk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk\" (UID: \"b8dbc896-b34b-4e32-a7ad-5155d9d10536\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"
Jul 14 17:36:03.985771 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:03.985637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8dbc896-b34b-4e32-a7ad-5155d9d10536-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk\" (UID: \"b8dbc896-b34b-4e32-a7ad-5155d9d10536\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"
Jul 14 17:36:04.086084 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:04.086050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8dbc896-b34b-4e32-a7ad-5155d9d10536-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk\" (UID: \"b8dbc896-b34b-4e32-a7ad-5155d9d10536\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"
Jul 14 17:36:04.086254 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:04.086101    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hhjk\" (UniqueName: \"kubernetes.io/projected/b8dbc896-b34b-4e32-a7ad-5155d9d10536-kube-api-access-7hhjk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk\" (UID: \"b8dbc896-b34b-4e32-a7ad-5155d9d10536\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"
Jul 14 17:36:04.086254 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:04.086137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8dbc896-b34b-4e32-a7ad-5155d9d10536-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk\" (UID: \"b8dbc896-b34b-4e32-a7ad-5155d9d10536\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"
Jul 14 17:36:04.086446 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:04.086427    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8dbc896-b34b-4e32-a7ad-5155d9d10536-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk\" (UID: \"b8dbc896-b34b-4e32-a7ad-5155d9d10536\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"
Jul 14 17:36:04.086521 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:04.086502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8dbc896-b34b-4e32-a7ad-5155d9d10536-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk\" (UID: \"b8dbc896-b34b-4e32-a7ad-5155d9d10536\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"
Jul 14 17:36:04.102629 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:04.102599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hhjk\" (UniqueName: \"kubernetes.io/projected/b8dbc896-b34b-4e32-a7ad-5155d9d10536-kube-api-access-7hhjk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk\" (UID: \"b8dbc896-b34b-4e32-a7ad-5155d9d10536\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"
Jul 14 17:36:04.236494 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:04.236402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"
Jul 14 17:36:04.372058 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:04.372034    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"]
Jul 14 17:36:04.374094 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:36:04.374065    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dbc896_b34b_4e32_a7ad_5155d9d10536.slice/crio-6e02b5a681cd9dd692875b52cdfb76a73f386457b9d6c346659b9aee3c893fc0 WatchSource:0}: Error finding container 6e02b5a681cd9dd692875b52cdfb76a73f386457b9d6c346659b9aee3c893fc0: Status 404 returned error can't find the container with id 6e02b5a681cd9dd692875b52cdfb76a73f386457b9d6c346659b9aee3c893fc0
Jul 14 17:36:04.664299 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:04.664193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk" event={"ID":"b8dbc896-b34b-4e32-a7ad-5155d9d10536","Type":"ContainerStarted","Data":"6e02b5a681cd9dd692875b52cdfb76a73f386457b9d6c346659b9aee3c893fc0"}
Jul 14 17:36:11.692448 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:11.692406    2578 generic.go:356] "Generic (PLEG): container finished" podID="b8dbc896-b34b-4e32-a7ad-5155d9d10536" containerID="9395e4fda2db9644ab754cbca964426a27efffb6b45582cc1d5c0245d781335d" exitCode=0
Jul 14 17:36:11.692842 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:11.692499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk" event={"ID":"b8dbc896-b34b-4e32-a7ad-5155d9d10536","Type":"ContainerDied","Data":"9395e4fda2db9644ab754cbca964426a27efffb6b45582cc1d5c0245d781335d"}
Jul 14 17:36:22.733808 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:22.733767    2578 generic.go:356] "Generic (PLEG): container finished" podID="b8dbc896-b34b-4e32-a7ad-5155d9d10536" containerID="e235ee9463270ae88b532bd9e1b14ef81d4e5b3921090c93f2526fa4849a74e2" exitCode=0
Jul 14 17:36:22.734364 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:22.733813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk" event={"ID":"b8dbc896-b34b-4e32-a7ad-5155d9d10536","Type":"ContainerDied","Data":"e235ee9463270ae88b532bd9e1b14ef81d4e5b3921090c93f2526fa4849a74e2"}
Jul 14 17:36:30.764913 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:30.764879    2578 generic.go:356] "Generic (PLEG): container finished" podID="b8dbc896-b34b-4e32-a7ad-5155d9d10536" containerID="2830a7092ac60c145fb3cf8a6120105d47def6936c28b6c074c4b83818e254d1" exitCode=0
Jul 14 17:36:30.765352 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:30.764947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk" event={"ID":"b8dbc896-b34b-4e32-a7ad-5155d9d10536","Type":"ContainerDied","Data":"2830a7092ac60c145fb3cf8a6120105d47def6936c28b6c074c4b83818e254d1"}
Jul 14 17:36:31.891285 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:31.891260    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"
Jul 14 17:36:32.007856 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:32.007825    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hhjk\" (UniqueName: \"kubernetes.io/projected/b8dbc896-b34b-4e32-a7ad-5155d9d10536-kube-api-access-7hhjk\") pod \"b8dbc896-b34b-4e32-a7ad-5155d9d10536\" (UID: \"b8dbc896-b34b-4e32-a7ad-5155d9d10536\") "
Jul 14 17:36:32.008055 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:32.007887    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8dbc896-b34b-4e32-a7ad-5155d9d10536-bundle\") pod \"b8dbc896-b34b-4e32-a7ad-5155d9d10536\" (UID: \"b8dbc896-b34b-4e32-a7ad-5155d9d10536\") "
Jul 14 17:36:32.008055 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:32.007926    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8dbc896-b34b-4e32-a7ad-5155d9d10536-util\") pod \"b8dbc896-b34b-4e32-a7ad-5155d9d10536\" (UID: \"b8dbc896-b34b-4e32-a7ad-5155d9d10536\") "
Jul 14 17:36:32.008599 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:32.008575    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8dbc896-b34b-4e32-a7ad-5155d9d10536-bundle" (OuterVolumeSpecName: "bundle") pod "b8dbc896-b34b-4e32-a7ad-5155d9d10536" (UID: "b8dbc896-b34b-4e32-a7ad-5155d9d10536"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:36:32.010255 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:32.010220    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8dbc896-b34b-4e32-a7ad-5155d9d10536-kube-api-access-7hhjk" (OuterVolumeSpecName: "kube-api-access-7hhjk") pod "b8dbc896-b34b-4e32-a7ad-5155d9d10536" (UID: "b8dbc896-b34b-4e32-a7ad-5155d9d10536"). InnerVolumeSpecName "kube-api-access-7hhjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:36:32.013004 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:32.012981    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8dbc896-b34b-4e32-a7ad-5155d9d10536-util" (OuterVolumeSpecName: "util") pod "b8dbc896-b34b-4e32-a7ad-5155d9d10536" (UID: "b8dbc896-b34b-4e32-a7ad-5155d9d10536"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:36:32.109025 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:32.108941    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8dbc896-b34b-4e32-a7ad-5155d9d10536-bundle\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:36:32.109025 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:32.108971    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8dbc896-b34b-4e32-a7ad-5155d9d10536-util\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:36:32.109025 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:32.108979    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hhjk\" (UniqueName: \"kubernetes.io/projected/b8dbc896-b34b-4e32-a7ad-5155d9d10536-kube-api-access-7hhjk\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:36:32.773820 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:32.773787    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk" event={"ID":"b8dbc896-b34b-4e32-a7ad-5155d9d10536","Type":"ContainerDied","Data":"6e02b5a681cd9dd692875b52cdfb76a73f386457b9d6c346659b9aee3c893fc0"}
Jul 14 17:36:32.773820 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:32.773810    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkbgk"
Jul 14 17:36:32.774014 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:32.773834    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e02b5a681cd9dd692875b52cdfb76a73f386457b9d6c346659b9aee3c893fc0"
Jul 14 17:36:35.299353 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:35.299320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:35.836845 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:35.836806    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:36.999606 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:36.999571    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:37.860941 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:37.860904    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kk2vf"]
Jul 14 17:36:37.861229 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:37.861193    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dbc896-b34b-4e32-a7ad-5155d9d10536" containerName="extract"
Jul 14 17:36:37.861229 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:37.861227    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dbc896-b34b-4e32-a7ad-5155d9d10536" containerName="extract"
Jul 14 17:36:37.861375 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:37.861248    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dbc896-b34b-4e32-a7ad-5155d9d10536" containerName="util"
Jul 14 17:36:37.861375 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:37.861256    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dbc896-b34b-4e32-a7ad-5155d9d10536" containerName="util"
Jul 14 17:36:37.861375 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:37.861275    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dbc896-b34b-4e32-a7ad-5155d9d10536" containerName="pull"
Jul 14 17:36:37.861375 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:37.861282    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dbc896-b34b-4e32-a7ad-5155d9d10536" containerName="pull"
Jul 14 17:36:37.861375 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:37.861350    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8dbc896-b34b-4e32-a7ad-5155d9d10536" containerName="extract"
Jul 14 17:36:37.863287 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:37.863267    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kk2vf"
Jul 14 17:36:37.866097 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:37.866078    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 14 17:36:37.866193 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:37.866079    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 14 17:36:37.866193 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:37.866160    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xmpnp\""
Jul 14 17:36:37.903059 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:37.903027    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kk2vf"]
Jul 14 17:36:37.952970 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:37.952929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmb49\" (UniqueName: \"kubernetes.io/projected/88658a12-a5bf-489d-9c4e-4f3f7646364f-kube-api-access-mmb49\") pod \"cert-manager-operator-controller-manager-55c444bf86-kk2vf\" (UID: \"88658a12-a5bf-489d-9c4e-4f3f7646364f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kk2vf"
Jul 14 17:36:37.953143 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:37.953056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88658a12-a5bf-489d-9c4e-4f3f7646364f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kk2vf\" (UID: \"88658a12-a5bf-489d-9c4e-4f3f7646364f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kk2vf"
Jul 14 17:36:38.054129 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:38.054089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88658a12-a5bf-489d-9c4e-4f3f7646364f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kk2vf\" (UID: \"88658a12-a5bf-489d-9c4e-4f3f7646364f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kk2vf"
Jul 14 17:36:38.054129 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:38.054129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmb49\" (UniqueName: \"kubernetes.io/projected/88658a12-a5bf-489d-9c4e-4f3f7646364f-kube-api-access-mmb49\") pod \"cert-manager-operator-controller-manager-55c444bf86-kk2vf\" (UID: \"88658a12-a5bf-489d-9c4e-4f3f7646364f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kk2vf"
Jul 14 17:36:38.054604 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:38.054479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88658a12-a5bf-489d-9c4e-4f3f7646364f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kk2vf\" (UID: \"88658a12-a5bf-489d-9c4e-4f3f7646364f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kk2vf"
Jul 14 17:36:38.066780 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:38.066755    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmb49\" (UniqueName: \"kubernetes.io/projected/88658a12-a5bf-489d-9c4e-4f3f7646364f-kube-api-access-mmb49\") pod \"cert-manager-operator-controller-manager-55c444bf86-kk2vf\" (UID: \"88658a12-a5bf-489d-9c4e-4f3f7646364f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kk2vf"
Jul 14 17:36:38.172241 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:38.172193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kk2vf"
Jul 14 17:36:38.357533 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:38.357504    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kk2vf"]
Jul 14 17:36:38.360995 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:36:38.360964    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88658a12_a5bf_489d_9c4e_4f3f7646364f.slice/crio-04d40c9d875147cbe3c6be3d6a1a5448c3b8d2ad5ea07a2ac9c1a3a3c008dd88 WatchSource:0}: Error finding container 04d40c9d875147cbe3c6be3d6a1a5448c3b8d2ad5ea07a2ac9c1a3a3c008dd88: Status 404 returned error can't find the container with id 04d40c9d875147cbe3c6be3d6a1a5448c3b8d2ad5ea07a2ac9c1a3a3c008dd88
Jul 14 17:36:38.794299 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:38.794263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kk2vf" event={"ID":"88658a12-a5bf-489d-9c4e-4f3f7646364f","Type":"ContainerStarted","Data":"04d40c9d875147cbe3c6be3d6a1a5448c3b8d2ad5ea07a2ac9c1a3a3c008dd88"}
Jul 14 17:36:39.512742 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:39.512710    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:39.614816 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:39.614770    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:40.803623 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:40.803584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kk2vf" event={"ID":"88658a12-a5bf-489d-9c4e-4f3f7646364f","Type":"ContainerStarted","Data":"83c919f268001d2de1d3f12f1c77374d1563a3d988c5c111c17af3263ecb0ed7"}
Jul 14 17:36:40.833720 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:40.833668    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kk2vf" podStartSLOduration=2.306768114 podStartE2EDuration="3.833650788s" totalImagesPullingTime="1.526882674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:36:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:36:40.832762287 +0000 UTC m=+257.549155980" watchObservedRunningTime="2026-07-14 17:36:40.833650788 +0000 UTC m=+257.550044479"
Jul 14 17:36:41.294387 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.294353    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:41.435502 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.435451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:41.497237 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.497191    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"]
Jul 14 17:36:41.499434 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.499416    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"
Jul 14 17:36:41.503029 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.503010    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 14 17:36:41.503171 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.503154    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zdg2s\""
Jul 14 17:36:41.503913 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.503896    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 14 17:36:41.515149 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.515127    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"]
Jul 14 17:36:41.584851 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.584771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8f8abe1-7ce2-4810-ae16-25d1048155a4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr\" (UID: \"b8f8abe1-7ce2-4810-ae16-25d1048155a4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"
Jul 14 17:36:41.584851 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.584819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8f8abe1-7ce2-4810-ae16-25d1048155a4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr\" (UID: \"b8f8abe1-7ce2-4810-ae16-25d1048155a4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"
Jul 14 17:36:41.584851 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.584845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h4rx\" (UniqueName: \"kubernetes.io/projected/b8f8abe1-7ce2-4810-ae16-25d1048155a4-kube-api-access-2h4rx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr\" (UID: \"b8f8abe1-7ce2-4810-ae16-25d1048155a4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"
Jul 14 17:36:41.662410 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.662357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:41.685467 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.685436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8f8abe1-7ce2-4810-ae16-25d1048155a4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr\" (UID: \"b8f8abe1-7ce2-4810-ae16-25d1048155a4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"
Jul 14 17:36:41.685607 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.685481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8f8abe1-7ce2-4810-ae16-25d1048155a4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr\" (UID: \"b8f8abe1-7ce2-4810-ae16-25d1048155a4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"
Jul 14 17:36:41.685607 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.685509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h4rx\" (UniqueName: \"kubernetes.io/projected/b8f8abe1-7ce2-4810-ae16-25d1048155a4-kube-api-access-2h4rx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr\" (UID: \"b8f8abe1-7ce2-4810-ae16-25d1048155a4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"
Jul 14 17:36:41.685801 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.685783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8f8abe1-7ce2-4810-ae16-25d1048155a4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr\" (UID: \"b8f8abe1-7ce2-4810-ae16-25d1048155a4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"
Jul 14 17:36:41.685835 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.685821    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8f8abe1-7ce2-4810-ae16-25d1048155a4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr\" (UID: \"b8f8abe1-7ce2-4810-ae16-25d1048155a4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"
Jul 14 17:36:41.700723 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.700694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h4rx\" (UniqueName: \"kubernetes.io/projected/b8f8abe1-7ce2-4810-ae16-25d1048155a4-kube-api-access-2h4rx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr\" (UID: \"b8f8abe1-7ce2-4810-ae16-25d1048155a4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"
Jul 14 17:36:41.812400 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.812374    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"
Jul 14 17:36:41.837095 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.837031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:41.947690 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:41.947666    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"]
Jul 14 17:36:41.949667 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:36:41.949642    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f8abe1_7ce2_4810_ae16_25d1048155a4.slice/crio-0337eaa38b156db690453a8c13828afec31079f90563790f462f8b3d087c4c57 WatchSource:0}: Error finding container 0337eaa38b156db690453a8c13828afec31079f90563790f462f8b3d087c4c57: Status 404 returned error can't find the container with id 0337eaa38b156db690453a8c13828afec31079f90563790f462f8b3d087c4c57
Jul 14 17:36:42.812110 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:42.812065    2578 generic.go:356] "Generic (PLEG): container finished" podID="b8f8abe1-7ce2-4810-ae16-25d1048155a4" containerID="04b3449805813d97d434a6be0bacbdc139111ecf7a31e10e55e861acd7be089c" exitCode=0
Jul 14 17:36:42.812289 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:42.812151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr" event={"ID":"b8f8abe1-7ce2-4810-ae16-25d1048155a4","Type":"ContainerDied","Data":"04b3449805813d97d434a6be0bacbdc139111ecf7a31e10e55e861acd7be089c"}
Jul 14 17:36:42.812289 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:42.812222    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr" event={"ID":"b8f8abe1-7ce2-4810-ae16-25d1048155a4","Type":"ContainerStarted","Data":"0337eaa38b156db690453a8c13828afec31079f90563790f462f8b3d087c4c57"}
Jul 14 17:36:43.035595 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:43.035509    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:44.064352 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:44.064272    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:44.266131 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:44.266087    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:45.486116 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.486086    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-v968l"]
Jul 14 17:36:45.488491 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.488474    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-v968l"
Jul 14 17:36:45.492686 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.492659    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 14 17:36:45.492798 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.492782    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-bz499\""
Jul 14 17:36:45.493677 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.493659    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 14 17:36:45.512763 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.512740    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-v968l"]
Jul 14 17:36:45.620021 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.619982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/866e0ccf-7f60-4514-a115-f9bd856cb530-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-v968l\" (UID: \"866e0ccf-7f60-4514-a115-f9bd856cb530\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v968l"
Jul 14 17:36:45.620233 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.620076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jkmv\" (UniqueName: \"kubernetes.io/projected/866e0ccf-7f60-4514-a115-f9bd856cb530-kube-api-access-9jkmv\") pod \"cert-manager-cainjector-559cdd8c87-v968l\" (UID: \"866e0ccf-7f60-4514-a115-f9bd856cb530\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v968l"
Jul 14 17:36:45.720917 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.720885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/866e0ccf-7f60-4514-a115-f9bd856cb530-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-v968l\" (UID: \"866e0ccf-7f60-4514-a115-f9bd856cb530\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v968l"
Jul 14 17:36:45.721087 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.720953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jkmv\" (UniqueName: \"kubernetes.io/projected/866e0ccf-7f60-4514-a115-f9bd856cb530-kube-api-access-9jkmv\") pod \"cert-manager-cainjector-559cdd8c87-v968l\" (UID: \"866e0ccf-7f60-4514-a115-f9bd856cb530\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v968l"
Jul 14 17:36:45.740996 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.740925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/866e0ccf-7f60-4514-a115-f9bd856cb530-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-v968l\" (UID: \"866e0ccf-7f60-4514-a115-f9bd856cb530\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v968l"
Jul 14 17:36:45.742884 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.742858    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jkmv\" (UniqueName: \"kubernetes.io/projected/866e0ccf-7f60-4514-a115-f9bd856cb530-kube-api-access-9jkmv\") pod \"cert-manager-cainjector-559cdd8c87-v968l\" (UID: \"866e0ccf-7f60-4514-a115-f9bd856cb530\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v968l"
Jul 14 17:36:45.807667 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.807634    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-v968l"
Jul 14 17:36:45.827371 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.827341    2578 generic.go:356] "Generic (PLEG): container finished" podID="b8f8abe1-7ce2-4810-ae16-25d1048155a4" containerID="db92e311271a660bd858cadd3d648da6e7826eee38690f0d188cef9b22201ce0" exitCode=0
Jul 14 17:36:45.827515 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.827421    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr" event={"ID":"b8f8abe1-7ce2-4810-ae16-25d1048155a4","Type":"ContainerDied","Data":"db92e311271a660bd858cadd3d648da6e7826eee38690f0d188cef9b22201ce0"}
Jul 14 17:36:45.967220 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:45.967023    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-v968l"]
Jul 14 17:36:45.968623 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:36:45.968591    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod866e0ccf_7f60_4514_a115_f9bd856cb530.slice/crio-68004935e2bdf6881574d79ea4a6e2b771e2c437041a4e17b851f5d69b12f332 WatchSource:0}: Error finding container 68004935e2bdf6881574d79ea4a6e2b771e2c437041a4e17b851f5d69b12f332: Status 404 returned error can't find the container with id 68004935e2bdf6881574d79ea4a6e2b771e2c437041a4e17b851f5d69b12f332
Jul 14 17:36:46.005849 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:46.005825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:46.150560 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:46.150507    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:46.833855 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:46.833822    2578 generic.go:356] "Generic (PLEG): container finished" podID="b8f8abe1-7ce2-4810-ae16-25d1048155a4" containerID="2a0b825ea934d8988171a4e0fe7177cc9f2137d6fe7840925f2423a789ec2b05" exitCode=0
Jul 14 17:36:46.834299 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:46.833915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr" event={"ID":"b8f8abe1-7ce2-4810-ae16-25d1048155a4","Type":"ContainerDied","Data":"2a0b825ea934d8988171a4e0fe7177cc9f2137d6fe7840925f2423a789ec2b05"}
Jul 14 17:36:46.835026 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:46.835008    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-v968l" event={"ID":"866e0ccf-7f60-4514-a115-f9bd856cb530","Type":"ContainerStarted","Data":"68004935e2bdf6881574d79ea4a6e2b771e2c437041a4e17b851f5d69b12f332"}
Jul 14 17:36:47.980743 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:47.980719    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"
Jul 14 17:36:48.142447 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:48.142350    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8f8abe1-7ce2-4810-ae16-25d1048155a4-util\") pod \"b8f8abe1-7ce2-4810-ae16-25d1048155a4\" (UID: \"b8f8abe1-7ce2-4810-ae16-25d1048155a4\") "
Jul 14 17:36:48.142447 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:48.142428    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8f8abe1-7ce2-4810-ae16-25d1048155a4-bundle\") pod \"b8f8abe1-7ce2-4810-ae16-25d1048155a4\" (UID: \"b8f8abe1-7ce2-4810-ae16-25d1048155a4\") "
Jul 14 17:36:48.142689 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:48.142497    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h4rx\" (UniqueName: \"kubernetes.io/projected/b8f8abe1-7ce2-4810-ae16-25d1048155a4-kube-api-access-2h4rx\") pod \"b8f8abe1-7ce2-4810-ae16-25d1048155a4\" (UID: \"b8f8abe1-7ce2-4810-ae16-25d1048155a4\") "
Jul 14 17:36:48.142852 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:48.142815    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f8abe1-7ce2-4810-ae16-25d1048155a4-bundle" (OuterVolumeSpecName: "bundle") pod "b8f8abe1-7ce2-4810-ae16-25d1048155a4" (UID: "b8f8abe1-7ce2-4810-ae16-25d1048155a4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:36:48.145071 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:48.145024    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f8abe1-7ce2-4810-ae16-25d1048155a4-kube-api-access-2h4rx" (OuterVolumeSpecName: "kube-api-access-2h4rx") pod "b8f8abe1-7ce2-4810-ae16-25d1048155a4" (UID: "b8f8abe1-7ce2-4810-ae16-25d1048155a4"). InnerVolumeSpecName "kube-api-access-2h4rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:36:48.149593 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:48.149563    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f8abe1-7ce2-4810-ae16-25d1048155a4-util" (OuterVolumeSpecName: "util") pod "b8f8abe1-7ce2-4810-ae16-25d1048155a4" (UID: "b8f8abe1-7ce2-4810-ae16-25d1048155a4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:36:48.243321 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:48.243279    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8f8abe1-7ce2-4810-ae16-25d1048155a4-util\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:36:48.243321 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:48.243318    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8f8abe1-7ce2-4810-ae16-25d1048155a4-bundle\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:36:48.243562 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:48.243333    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h4rx\" (UniqueName: \"kubernetes.io/projected/b8f8abe1-7ce2-4810-ae16-25d1048155a4-kube-api-access-2h4rx\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:36:48.849029 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:48.848991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr" event={"ID":"b8f8abe1-7ce2-4810-ae16-25d1048155a4","Type":"ContainerDied","Data":"0337eaa38b156db690453a8c13828afec31079f90563790f462f8b3d087c4c57"}
Jul 14 17:36:48.849196 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:48.849051    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0337eaa38b156db690453a8c13828afec31079f90563790f462f8b3d087c4c57"
Jul 14 17:36:48.849196 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:48.849013    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8jkjr"
Jul 14 17:36:49.855289 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:49.855251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-v968l" event={"ID":"866e0ccf-7f60-4514-a115-f9bd856cb530","Type":"ContainerStarted","Data":"ca0eeddbe60dbedd8f31dc6af4cf18d094fdfb845edbb06c1ea90c2d33d9925e"}
Jul 14 17:36:49.904018 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:49.903963    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-v968l" podStartSLOduration=1.498053837 podStartE2EDuration="4.903949388s" totalImagesPullingTime="3.405895551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:36:49.902056014 +0000 UTC m=+266.618449707" watchObservedRunningTime="2026-07-14 17:36:49.903949388 +0000 UTC m=+266.620343079"
Jul 14 17:36:50.228688 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.228651    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rrxk4"]
Jul 14 17:36:50.228921 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.228910    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f8abe1-7ce2-4810-ae16-25d1048155a4" containerName="extract"
Jul 14 17:36:50.228965 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.228923    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f8abe1-7ce2-4810-ae16-25d1048155a4" containerName="extract"
Jul 14 17:36:50.228965 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.228934    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f8abe1-7ce2-4810-ae16-25d1048155a4" containerName="pull"
Jul 14 17:36:50.228965 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.228940    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f8abe1-7ce2-4810-ae16-25d1048155a4" containerName="pull"
Jul 14 17:36:50.228965 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.228950    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f8abe1-7ce2-4810-ae16-25d1048155a4" containerName="util"
Jul 14 17:36:50.228965 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.228955    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f8abe1-7ce2-4810-ae16-25d1048155a4" containerName="util"
Jul 14 17:36:50.229108 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.228999    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8f8abe1-7ce2-4810-ae16-25d1048155a4" containerName="extract"
Jul 14 17:36:50.231930 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.231909    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rrxk4"
Jul 14 17:36:50.237500 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.237470    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-r29hp\""
Jul 14 17:36:50.255140 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.255114    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rrxk4"]
Jul 14 17:36:50.360607 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.360576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7583e053-7bb9-44c5-a04b-14fbf5072d81-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rrxk4\" (UID: \"7583e053-7bb9-44c5-a04b-14fbf5072d81\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rrxk4"
Jul 14 17:36:50.360790 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.360672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrfzj\" (UniqueName: \"kubernetes.io/projected/7583e053-7bb9-44c5-a04b-14fbf5072d81-kube-api-access-qrfzj\") pod \"cert-manager-webhook-6486ffd997-rrxk4\" (UID: \"7583e053-7bb9-44c5-a04b-14fbf5072d81\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rrxk4"
Jul 14 17:36:50.461324 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.461285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrfzj\" (UniqueName: \"kubernetes.io/projected/7583e053-7bb9-44c5-a04b-14fbf5072d81-kube-api-access-qrfzj\") pod \"cert-manager-webhook-6486ffd997-rrxk4\" (UID: \"7583e053-7bb9-44c5-a04b-14fbf5072d81\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rrxk4"
Jul 14 17:36:50.461497 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.461335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7583e053-7bb9-44c5-a04b-14fbf5072d81-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rrxk4\" (UID: \"7583e053-7bb9-44c5-a04b-14fbf5072d81\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rrxk4"
Jul 14 17:36:50.477060 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.477034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrfzj\" (UniqueName: \"kubernetes.io/projected/7583e053-7bb9-44c5-a04b-14fbf5072d81-kube-api-access-qrfzj\") pod \"cert-manager-webhook-6486ffd997-rrxk4\" (UID: \"7583e053-7bb9-44c5-a04b-14fbf5072d81\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rrxk4"
Jul 14 17:36:50.477459 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.477437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7583e053-7bb9-44c5-a04b-14fbf5072d81-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rrxk4\" (UID: \"7583e053-7bb9-44c5-a04b-14fbf5072d81\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rrxk4"
Jul 14 17:36:50.540916 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.540836    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rrxk4"
Jul 14 17:36:50.686828 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.686796    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rrxk4"]
Jul 14 17:36:50.688497 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:36:50.688469    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7583e053_7bb9_44c5_a04b_14fbf5072d81.slice/crio-38a9968cb20dd686eb87419b4fda4dd5c69e7e55173e90db54f0156d364305a2 WatchSource:0}: Error finding container 38a9968cb20dd686eb87419b4fda4dd5c69e7e55173e90db54f0156d364305a2: Status 404 returned error can't find the container with id 38a9968cb20dd686eb87419b4fda4dd5c69e7e55173e90db54f0156d364305a2
Jul 14 17:36:50.860596 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.860502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rrxk4" event={"ID":"7583e053-7bb9-44c5-a04b-14fbf5072d81","Type":"ContainerStarted","Data":"a23cb360d61621883732af0a69e880fe82e446ee584a6d99976819e2b1fbb723"}
Jul 14 17:36:50.860596 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.860553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rrxk4" event={"ID":"7583e053-7bb9-44c5-a04b-14fbf5072d81","Type":"ContainerStarted","Data":"38a9968cb20dd686eb87419b4fda4dd5c69e7e55173e90db54f0156d364305a2"}
Jul 14 17:36:50.897099 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:50.897042    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-rrxk4" podStartSLOduration=0.897027719 podStartE2EDuration="897.027719ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:36:50.895626945 +0000 UTC m=+267.612020660" watchObservedRunningTime="2026-07-14 17:36:50.897027719 +0000 UTC m=+267.613421411"
Jul 14 17:36:51.002529 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:51.002495    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:51.105038 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:51.105006    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:51.288246 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:51.288189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:51.865321 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:51.865289    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rrxk4"
Jul 14 17:36:53.519149 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:53.519116    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:53.628416 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:53.628383    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:36:54.944760 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:54.944723    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcdmf"]
Jul 14 17:36:54.957430 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:54.957402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcdmf"
Jul 14 17:36:54.958425 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:54.958398    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcdmf"]
Jul 14 17:36:54.960230 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:54.960196    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 14 17:36:54.960336 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:54.960196    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 14 17:36:54.961037 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:54.961018    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-rxllx\""
Jul 14 17:36:55.095374 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:55.095340    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f45c09af-344b-4666-a2d3-f8da53eea0bb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fcdmf\" (UID: \"f45c09af-344b-4666-a2d3-f8da53eea0bb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcdmf"
Jul 14 17:36:55.095548 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:55.095382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2dk\" (UniqueName: \"kubernetes.io/projected/f45c09af-344b-4666-a2d3-f8da53eea0bb-kube-api-access-9d2dk\") pod \"openshift-lws-operator-fd8ccff4c-fcdmf\" (UID: \"f45c09af-344b-4666-a2d3-f8da53eea0bb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcdmf"
Jul 14 17:36:55.196757 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:55.196685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f45c09af-344b-4666-a2d3-f8da53eea0bb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fcdmf\" (UID: \"f45c09af-344b-4666-a2d3-f8da53eea0bb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcdmf"
Jul 14 17:36:55.196757 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:55.196726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2dk\" (UniqueName: \"kubernetes.io/projected/f45c09af-344b-4666-a2d3-f8da53eea0bb-kube-api-access-9d2dk\") pod \"openshift-lws-operator-fd8ccff4c-fcdmf\" (UID: \"f45c09af-344b-4666-a2d3-f8da53eea0bb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcdmf"
Jul 14 17:36:55.197052 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:55.197034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f45c09af-344b-4666-a2d3-f8da53eea0bb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fcdmf\" (UID: \"f45c09af-344b-4666-a2d3-f8da53eea0bb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcdmf"
Jul 14 17:36:55.207127 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:55.207094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2dk\" (UniqueName: \"kubernetes.io/projected/f45c09af-344b-4666-a2d3-f8da53eea0bb-kube-api-access-9d2dk\") pod \"openshift-lws-operator-fd8ccff4c-fcdmf\" (UID: \"f45c09af-344b-4666-a2d3-f8da53eea0bb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcdmf"
Jul 14 17:36:55.267672 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:55.267634    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcdmf"
Jul 14 17:36:55.439184 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:55.439153    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcdmf"]
Jul 14 17:36:55.442436 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:36:55.442411    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf45c09af_344b_4666_a2d3_f8da53eea0bb.slice/crio-bb8ca26bee7243c6df52e84638d0c8a0ba031854e95d1034de4a22e8240bbc85 WatchSource:0}: Error finding container bb8ca26bee7243c6df52e84638d0c8a0ba031854e95d1034de4a22e8240bbc85: Status 404 returned error can't find the container with id bb8ca26bee7243c6df52e84638d0c8a0ba031854e95d1034de4a22e8240bbc85
Jul 14 17:36:55.884345 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:55.884266    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcdmf" event={"ID":"f45c09af-344b-4666-a2d3-f8da53eea0bb","Type":"ContainerStarted","Data":"bb8ca26bee7243c6df52e84638d0c8a0ba031854e95d1034de4a22e8240bbc85"}
Jul 14 17:36:57.872564 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:57.872540    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rrxk4"
Jul 14 17:36:58.897044 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:58.897007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcdmf" event={"ID":"f45c09af-344b-4666-a2d3-f8da53eea0bb","Type":"ContainerStarted","Data":"81077bc9a19444f3a581a663332c7873663c53e64ada03b1a35e4d16022857c0"}
Jul 14 17:36:58.920523 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:58.920477    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcdmf" podStartSLOduration=2.501109478 podStartE2EDuration="4.920459365s" totalImagesPullingTime="2.419349887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:36:58.919686333 +0000 UTC m=+275.636080031" watchObservedRunningTime="2026-07-14 17:36:58.920459365 +0000 UTC m=+275.636853057"
Jul 14 17:36:59.710127 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:36:59.710092    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:01.370160 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.370118    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"]
Jul 14 17:37:01.372721 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.372699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"
Jul 14 17:37:01.375996 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.375976    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 14 17:37:01.376782 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.376766    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zdg2s\""
Jul 14 17:37:01.377313 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.377287    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 14 17:37:01.389789 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.389765    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"]
Jul 14 17:37:01.545544 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.545502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvl9w\" (UniqueName: \"kubernetes.io/projected/f4a813b9-47c6-4eef-b07a-6df028c2ef78-kube-api-access-nvl9w\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg\" (UID: \"f4a813b9-47c6-4eef-b07a-6df028c2ef78\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"
Jul 14 17:37:01.545719 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.545549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4a813b9-47c6-4eef-b07a-6df028c2ef78-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg\" (UID: \"f4a813b9-47c6-4eef-b07a-6df028c2ef78\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"
Jul 14 17:37:01.545719 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.545668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4a813b9-47c6-4eef-b07a-6df028c2ef78-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg\" (UID: \"f4a813b9-47c6-4eef-b07a-6df028c2ef78\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"
Jul 14 17:37:01.647046 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.646945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4a813b9-47c6-4eef-b07a-6df028c2ef78-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg\" (UID: \"f4a813b9-47c6-4eef-b07a-6df028c2ef78\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"
Jul 14 17:37:01.647046 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.647004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvl9w\" (UniqueName: \"kubernetes.io/projected/f4a813b9-47c6-4eef-b07a-6df028c2ef78-kube-api-access-nvl9w\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg\" (UID: \"f4a813b9-47c6-4eef-b07a-6df028c2ef78\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"
Jul 14 17:37:01.647046 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.647024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4a813b9-47c6-4eef-b07a-6df028c2ef78-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg\" (UID: \"f4a813b9-47c6-4eef-b07a-6df028c2ef78\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"
Jul 14 17:37:01.647510 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.647489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4a813b9-47c6-4eef-b07a-6df028c2ef78-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg\" (UID: \"f4a813b9-47c6-4eef-b07a-6df028c2ef78\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"
Jul 14 17:37:01.647552 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.647509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4a813b9-47c6-4eef-b07a-6df028c2ef78-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg\" (UID: \"f4a813b9-47c6-4eef-b07a-6df028c2ef78\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"
Jul 14 17:37:01.662625 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.662596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvl9w\" (UniqueName: \"kubernetes.io/projected/f4a813b9-47c6-4eef-b07a-6df028c2ef78-kube-api-access-nvl9w\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg\" (UID: \"f4a813b9-47c6-4eef-b07a-6df028c2ef78\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"
Jul 14 17:37:01.681390 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.681364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"
Jul 14 17:37:01.847150 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.847121    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"]
Jul 14 17:37:01.910869 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:01.910838    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg" event={"ID":"f4a813b9-47c6-4eef-b07a-6df028c2ef78","Type":"ContainerStarted","Data":"3101950e4cebdbf096931b2483b5f1a16089ca4cfd964e429bd3b196d1b5a62f"}
Jul 14 17:37:02.915923 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:02.915886    2578 generic.go:356] "Generic (PLEG): container finished" podID="f4a813b9-47c6-4eef-b07a-6df028c2ef78" containerID="b4ff2e8fd6fc2639ea2d4432c973837b0bacfd5206956ab41c76b5b25de1f928" exitCode=0
Jul 14 17:37:02.916321 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:02.915948    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg" event={"ID":"f4a813b9-47c6-4eef-b07a-6df028c2ef78","Type":"ContainerDied","Data":"b4ff2e8fd6fc2639ea2d4432c973837b0bacfd5206956ab41c76b5b25de1f928"}
Jul 14 17:37:03.547076 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:03.547046    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:03.922545 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:03.922510    2578 generic.go:356] "Generic (PLEG): container finished" podID="f4a813b9-47c6-4eef-b07a-6df028c2ef78" containerID="ea151cbe91677809413fa0573033760b09c0c5114b9bfcdc106a0cc6f45f6230" exitCode=0
Jul 14 17:37:03.922935 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:03.922558    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg" event={"ID":"f4a813b9-47c6-4eef-b07a-6df028c2ef78","Type":"ContainerDied","Data":"ea151cbe91677809413fa0573033760b09c0c5114b9bfcdc106a0cc6f45f6230"}
Jul 14 17:37:04.929593 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:04.929559    2578 generic.go:356] "Generic (PLEG): container finished" podID="f4a813b9-47c6-4eef-b07a-6df028c2ef78" containerID="d30ff06d9b7797c288faa9c51caddd077200dbe4e4f3d71da1b886f4129baae2" exitCode=0
Jul 14 17:37:04.929958 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:04.929648    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg" event={"ID":"f4a813b9-47c6-4eef-b07a-6df028c2ef78","Type":"ContainerDied","Data":"d30ff06d9b7797c288faa9c51caddd077200dbe4e4f3d71da1b886f4129baae2"}
Jul 14 17:37:06.053785 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:06.053760    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"
Jul 14 17:37:06.076394 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:06.076365    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4a813b9-47c6-4eef-b07a-6df028c2ef78-bundle\") pod \"f4a813b9-47c6-4eef-b07a-6df028c2ef78\" (UID: \"f4a813b9-47c6-4eef-b07a-6df028c2ef78\") "
Jul 14 17:37:06.076544 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:06.076418    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvl9w\" (UniqueName: \"kubernetes.io/projected/f4a813b9-47c6-4eef-b07a-6df028c2ef78-kube-api-access-nvl9w\") pod \"f4a813b9-47c6-4eef-b07a-6df028c2ef78\" (UID: \"f4a813b9-47c6-4eef-b07a-6df028c2ef78\") "
Jul 14 17:37:06.076544 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:06.076476    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4a813b9-47c6-4eef-b07a-6df028c2ef78-util\") pod \"f4a813b9-47c6-4eef-b07a-6df028c2ef78\" (UID: \"f4a813b9-47c6-4eef-b07a-6df028c2ef78\") "
Jul 14 17:37:06.077115 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:06.077089    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a813b9-47c6-4eef-b07a-6df028c2ef78-bundle" (OuterVolumeSpecName: "bundle") pod "f4a813b9-47c6-4eef-b07a-6df028c2ef78" (UID: "f4a813b9-47c6-4eef-b07a-6df028c2ef78"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:37:06.078703 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:06.078680    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a813b9-47c6-4eef-b07a-6df028c2ef78-kube-api-access-nvl9w" (OuterVolumeSpecName: "kube-api-access-nvl9w") pod "f4a813b9-47c6-4eef-b07a-6df028c2ef78" (UID: "f4a813b9-47c6-4eef-b07a-6df028c2ef78"). InnerVolumeSpecName "kube-api-access-nvl9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:37:06.082845 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:06.082811    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a813b9-47c6-4eef-b07a-6df028c2ef78-util" (OuterVolumeSpecName: "util") pod "f4a813b9-47c6-4eef-b07a-6df028c2ef78" (UID: "f4a813b9-47c6-4eef-b07a-6df028c2ef78"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:37:06.177164 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:06.177132    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvl9w\" (UniqueName: \"kubernetes.io/projected/f4a813b9-47c6-4eef-b07a-6df028c2ef78-kube-api-access-nvl9w\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:37:06.177164 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:06.177163    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4a813b9-47c6-4eef-b07a-6df028c2ef78-util\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:37:06.177164 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:06.177173    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4a813b9-47c6-4eef-b07a-6df028c2ef78-bundle\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:37:06.939737 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:06.939714    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg"
Jul 14 17:37:06.939894 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:06.939709    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgfcqg" event={"ID":"f4a813b9-47c6-4eef-b07a-6df028c2ef78","Type":"ContainerDied","Data":"3101950e4cebdbf096931b2483b5f1a16089ca4cfd964e429bd3b196d1b5a62f"}
Jul 14 17:37:06.939894 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:06.939834    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3101950e4cebdbf096931b2483b5f1a16089ca4cfd964e429bd3b196d1b5a62f"
Jul 14 17:37:17.817165 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:17.817132    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:18.042920 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:18.042886    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:18.584264 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:18.584232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:18.964974 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:18.964944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:19.076475 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.076439    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:19.101633 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.101597    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"]
Jul 14 17:37:19.101899 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.101886    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a813b9-47c6-4eef-b07a-6df028c2ef78" containerName="extract"
Jul 14 17:37:19.101940 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.101904    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a813b9-47c6-4eef-b07a-6df028c2ef78" containerName="extract"
Jul 14 17:37:19.101940 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.101917    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a813b9-47c6-4eef-b07a-6df028c2ef78" containerName="pull"
Jul 14 17:37:19.101940 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.101923    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a813b9-47c6-4eef-b07a-6df028c2ef78" containerName="pull"
Jul 14 17:37:19.101940 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.101936    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a813b9-47c6-4eef-b07a-6df028c2ef78" containerName="util"
Jul 14 17:37:19.102073 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.101941    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a813b9-47c6-4eef-b07a-6df028c2ef78" containerName="util"
Jul 14 17:37:19.102073 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.101991    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a813b9-47c6-4eef-b07a-6df028c2ef78" containerName="extract"
Jul 14 17:37:19.103858 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.103842    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.109118 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.106496    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 14 17:37:19.109118 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.107276    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 14 17:37:19.109118 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.107320    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xk54r\""
Jul 14 17:37:19.109118 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.107585    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 14 17:37:19.109118 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.107796    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 14 17:37:19.172855 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.172815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjkh8\" (UniqueName: \"kubernetes.io/projected/e41e0738-0f4b-4e94-bf50-a2482c47a957-kube-api-access-fjkh8\") pod \"opendatahub-operator-controller-manager-7558c48c44-hlgbh\" (UID: \"e41e0738-0f4b-4e94-bf50-a2482c47a957\") " pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.172855 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.172858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e41e0738-0f4b-4e94-bf50-a2482c47a957-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7558c48c44-hlgbh\" (UID: \"e41e0738-0f4b-4e94-bf50-a2482c47a957\") " pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.173036 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.172886    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e41e0738-0f4b-4e94-bf50-a2482c47a957-webhook-cert\") pod \"opendatahub-operator-controller-manager-7558c48c44-hlgbh\" (UID: \"e41e0738-0f4b-4e94-bf50-a2482c47a957\") " pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.173036 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.172906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e41e0738-0f4b-4e94-bf50-a2482c47a957-tmp\") pod \"opendatahub-operator-controller-manager-7558c48c44-hlgbh\" (UID: \"e41e0738-0f4b-4e94-bf50-a2482c47a957\") " pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.173036 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.172959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e41e0738-0f4b-4e94-bf50-a2482c47a957-manifests\") pod \"opendatahub-operator-controller-manager-7558c48c44-hlgbh\" (UID: \"e41e0738-0f4b-4e94-bf50-a2482c47a957\") " pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.182617 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.182588    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"]
Jul 14 17:37:19.273411 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.273323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjkh8\" (UniqueName: \"kubernetes.io/projected/e41e0738-0f4b-4e94-bf50-a2482c47a957-kube-api-access-fjkh8\") pod \"opendatahub-operator-controller-manager-7558c48c44-hlgbh\" (UID: \"e41e0738-0f4b-4e94-bf50-a2482c47a957\") " pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.273411 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.273358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e41e0738-0f4b-4e94-bf50-a2482c47a957-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7558c48c44-hlgbh\" (UID: \"e41e0738-0f4b-4e94-bf50-a2482c47a957\") " pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.273411 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.273388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e41e0738-0f4b-4e94-bf50-a2482c47a957-webhook-cert\") pod \"opendatahub-operator-controller-manager-7558c48c44-hlgbh\" (UID: \"e41e0738-0f4b-4e94-bf50-a2482c47a957\") " pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.273411 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.273409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e41e0738-0f4b-4e94-bf50-a2482c47a957-tmp\") pod \"opendatahub-operator-controller-manager-7558c48c44-hlgbh\" (UID: \"e41e0738-0f4b-4e94-bf50-a2482c47a957\") " pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.273732 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.273434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e41e0738-0f4b-4e94-bf50-a2482c47a957-manifests\") pod \"opendatahub-operator-controller-manager-7558c48c44-hlgbh\" (UID: \"e41e0738-0f4b-4e94-bf50-a2482c47a957\") " pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.273786 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.273768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e41e0738-0f4b-4e94-bf50-a2482c47a957-manifests\") pod \"opendatahub-operator-controller-manager-7558c48c44-hlgbh\" (UID: \"e41e0738-0f4b-4e94-bf50-a2482c47a957\") " pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.276016 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.275981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e41e0738-0f4b-4e94-bf50-a2482c47a957-tmp\") pod \"opendatahub-operator-controller-manager-7558c48c44-hlgbh\" (UID: \"e41e0738-0f4b-4e94-bf50-a2482c47a957\") " pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.276136 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.276064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e41e0738-0f4b-4e94-bf50-a2482c47a957-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7558c48c44-hlgbh\" (UID: \"e41e0738-0f4b-4e94-bf50-a2482c47a957\") " pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.276248 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.276229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e41e0738-0f4b-4e94-bf50-a2482c47a957-webhook-cert\") pod \"opendatahub-operator-controller-manager-7558c48c44-hlgbh\" (UID: \"e41e0738-0f4b-4e94-bf50-a2482c47a957\") " pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.285154 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.285129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjkh8\" (UniqueName: \"kubernetes.io/projected/e41e0738-0f4b-4e94-bf50-a2482c47a957-kube-api-access-fjkh8\") pod \"opendatahub-operator-controller-manager-7558c48c44-hlgbh\" (UID: \"e41e0738-0f4b-4e94-bf50-a2482c47a957\") " pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.418387 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.418348    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:19.574196 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.574116    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"]
Jul 14 17:37:19.578337 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:37:19.578312    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41e0738_0f4b_4e94_bf50_a2482c47a957.slice/crio-f0c76eb36357223694a7f07155b0a2b88d740f75f1fdd7ac3c82df878f739ac3 WatchSource:0}: Error finding container f0c76eb36357223694a7f07155b0a2b88d740f75f1fdd7ac3c82df878f739ac3: Status 404 returned error can't find the container with id f0c76eb36357223694a7f07155b0a2b88d740f75f1fdd7ac3c82df878f739ac3
Jul 14 17:37:19.620882 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.620854    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:19.726995 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.726962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:19.819298 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.819266    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:19.950348 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.950322    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:19.995224 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:19.995170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh" event={"ID":"e41e0738-0f4b-4e94-bf50-a2482c47a957","Type":"ContainerStarted","Data":"f0c76eb36357223694a7f07155b0a2b88d740f75f1fdd7ac3c82df878f739ac3"}
Jul 14 17:37:20.064066 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:20.064038    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:20.155226 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:20.155171    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:20.275661 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:20.275569    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:20.386672 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:20.386641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:20.505947 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:20.505918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:24.111242 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:24.111192    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 17:37:24.113495 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:24.111297    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 17:37:24.117988 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:24.117970    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 14 17:37:25.021031 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:25.020997    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh" event={"ID":"e41e0738-0f4b-4e94-bf50-a2482c47a957","Type":"ContainerStarted","Data":"26a9a713da6b0aa1946d5cdcb05eb2f9d3c1b74ee0a05a7fbf3975f4a65c7ada"}
Jul 14 17:37:25.806514 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:25.806481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:25.933063 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:25.933032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:26.026973 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:26.026941    2578 generic.go:356] "Generic (PLEG): container finished" podID="e41e0738-0f4b-4e94-bf50-a2482c47a957" containerID="26a9a713da6b0aa1946d5cdcb05eb2f9d3c1b74ee0a05a7fbf3975f4a65c7ada" exitCode=0
Jul 14 17:37:26.027149 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:26.027004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh" event={"ID":"e41e0738-0f4b-4e94-bf50-a2482c47a957","Type":"ContainerDied","Data":"26a9a713da6b0aa1946d5cdcb05eb2f9d3c1b74ee0a05a7fbf3975f4a65c7ada"}
Jul 14 17:37:26.027537 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:26.027519    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 17:37:27.032786 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:27.032753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh" event={"ID":"e41e0738-0f4b-4e94-bf50-a2482c47a957","Type":"ContainerStarted","Data":"a24565293f8b97931385a1c78e1c60ac80ebfd80d7547a07f4d723de472c1128"}
Jul 14 17:37:27.033157 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:27.032944    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:27.072002 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:27.071956    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh" podStartSLOduration=2.034404852 podStartE2EDuration="8.07193985s" totalImagesPullingTime="4.975745678s" totalInitContainerRuntime="1.06178932s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:37:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:37:27.06767667 +0000 UTC m=+303.784070381" watchObservedRunningTime="2026-07-14 17:37:27.07193985 +0000 UTC m=+303.788333542"
Jul 14 17:37:30.935997 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:30.935960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:31.029297 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:31.029263    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:38.040729 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:38.040700    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7558c48c44-hlgbh"
Jul 14 17:37:39.808953 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:39.808923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:39.923301 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:39.923253    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:40.479520 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:40.479485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:40.695595 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:40.695562    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv"]
Jul 14 17:37:40.701218 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:40.701184    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv"
Jul 14 17:37:40.704796 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:37:40.704770    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-creds\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" type="*v1.Secret"
Jul 14 17:37:40.707144 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:40.707122    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 17:37:40.707259 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:37:40.707156    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-tls\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret"
Jul 14 17:37:40.707609 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:40.707369    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 17:37:40.731866 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:40.731805    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv"]
Jul 14 17:37:40.746019 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:40.745999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m546b\" (UniqueName: \"kubernetes.io/projected/cd61ccf1-5789-425f-b22b-77db95bd220b-kube-api-access-m546b\") pod \"kube-auth-proxy-fd75767d5-j5vhv\" (UID: \"cd61ccf1-5789-425f-b22b-77db95bd220b\") " pod="openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv"
Jul 14 17:37:40.746098 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:40.746050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd61ccf1-5789-425f-b22b-77db95bd220b-tls-certs\") pod \"kube-auth-proxy-fd75767d5-j5vhv\" (UID: \"cd61ccf1-5789-425f-b22b-77db95bd220b\") " pod="openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv"
Jul 14 17:37:40.746098 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:40.746070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd61ccf1-5789-425f-b22b-77db95bd220b-tmp\") pod \"kube-auth-proxy-fd75767d5-j5vhv\" (UID: \"cd61ccf1-5789-425f-b22b-77db95bd220b\") " pod="openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv"
Jul 14 17:37:40.846880 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:40.846843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd61ccf1-5789-425f-b22b-77db95bd220b-tls-certs\") pod \"kube-auth-proxy-fd75767d5-j5vhv\" (UID: \"cd61ccf1-5789-425f-b22b-77db95bd220b\") " pod="openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv"
Jul 14 17:37:40.846880 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:40.846882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd61ccf1-5789-425f-b22b-77db95bd220b-tmp\") pod \"kube-auth-proxy-fd75767d5-j5vhv\" (UID: \"cd61ccf1-5789-425f-b22b-77db95bd220b\") " pod="openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv"
Jul 14 17:37:40.847422 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:40.846934    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m546b\" (UniqueName: \"kubernetes.io/projected/cd61ccf1-5789-425f-b22b-77db95bd220b-kube-api-access-m546b\") pod \"kube-auth-proxy-fd75767d5-j5vhv\" (UID: \"cd61ccf1-5789-425f-b22b-77db95bd220b\") " pod="openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv"
Jul 14 17:37:40.849455 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:40.849432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd61ccf1-5789-425f-b22b-77db95bd220b-tmp\") pod \"kube-auth-proxy-fd75767d5-j5vhv\" (UID: \"cd61ccf1-5789-425f-b22b-77db95bd220b\") " pod="openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv"
Jul 14 17:37:40.857540 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:40.857512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m546b\" (UniqueName: \"kubernetes.io/projected/cd61ccf1-5789-425f-b22b-77db95bd220b-kube-api-access-m546b\") pod \"kube-auth-proxy-fd75767d5-j5vhv\" (UID: \"cd61ccf1-5789-425f-b22b-77db95bd220b\") " pod="openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv"
Jul 14 17:37:40.858373 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:40.858348    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:41.130128 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.130047    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-d9mq2"]
Jul 14 17:37:41.132322 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.132300    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-d9mq2"
Jul 14 17:37:41.136244 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.136220    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 14 17:37:41.136548 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.136525    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qnzf7\""
Jul 14 17:37:41.153364 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.153333    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-d9mq2"]
Jul 14 17:37:41.239434 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.239404    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:41.251104 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.251078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c29c7995-8119-4a2d-b7bb-0b233ac8e1c8-cert\") pod \"odh-model-controller-7998c67dc4-d9mq2\" (UID: \"c29c7995-8119-4a2d-b7bb-0b233ac8e1c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-d9mq2"
Jul 14 17:37:41.251255 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.251114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bxjp\" (UniqueName: \"kubernetes.io/projected/c29c7995-8119-4a2d-b7bb-0b233ac8e1c8-kube-api-access-2bxjp\") pod \"odh-model-controller-7998c67dc4-d9mq2\" (UID: \"c29c7995-8119-4a2d-b7bb-0b233ac8e1c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-d9mq2"
Jul 14 17:37:41.319660 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.319622    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:41.352162 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.352127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c29c7995-8119-4a2d-b7bb-0b233ac8e1c8-cert\") pod \"odh-model-controller-7998c67dc4-d9mq2\" (UID: \"c29c7995-8119-4a2d-b7bb-0b233ac8e1c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-d9mq2"
Jul 14 17:37:41.352162 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.352167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxjp\" (UniqueName: \"kubernetes.io/projected/c29c7995-8119-4a2d-b7bb-0b233ac8e1c8-kube-api-access-2bxjp\") pod \"odh-model-controller-7998c67dc4-d9mq2\" (UID: \"c29c7995-8119-4a2d-b7bb-0b233ac8e1c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-d9mq2"
Jul 14 17:37:41.352379 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:37:41.352305    2578 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 14 17:37:41.352414 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:37:41.352380    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c29c7995-8119-4a2d-b7bb-0b233ac8e1c8-cert podName:c29c7995-8119-4a2d-b7bb-0b233ac8e1c8 nodeName:}" failed. No retries permitted until 2026-07-14 17:37:41.852362575 +0000 UTC m=+318.568756244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c29c7995-8119-4a2d-b7bb-0b233ac8e1c8-cert") pod "odh-model-controller-7998c67dc4-d9mq2" (UID: "c29c7995-8119-4a2d-b7bb-0b233ac8e1c8") : secret "odh-model-controller-webhook-cert" not found
Jul 14 17:37:41.386184 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.386101    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bxjp\" (UniqueName: \"kubernetes.io/projected/c29c7995-8119-4a2d-b7bb-0b233ac8e1c8-kube-api-access-2bxjp\") pod \"odh-model-controller-7998c67dc4-d9mq2\" (UID: \"c29c7995-8119-4a2d-b7bb-0b233ac8e1c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-d9mq2"
Jul 14 17:37:41.422036 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.422007    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:41.539432 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.539396    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:41.847820 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:37:41.847785    2578 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: failed to sync secret cache: timed out waiting for the condition
Jul 14 17:37:41.848170 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:37:41.847862    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd61ccf1-5789-425f-b22b-77db95bd220b-tls-certs podName:cd61ccf1-5789-425f-b22b-77db95bd220b nodeName:}" failed. No retries permitted until 2026-07-14 17:37:42.347846388 +0000 UTC m=+319.064240058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cd61ccf1-5789-425f-b22b-77db95bd220b-tls-certs") pod "kube-auth-proxy-fd75767d5-j5vhv" (UID: "cd61ccf1-5789-425f-b22b-77db95bd220b") : failed to sync secret cache: timed out waiting for the condition
Jul 14 17:37:41.857363 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.857335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c29c7995-8119-4a2d-b7bb-0b233ac8e1c8-cert\") pod \"odh-model-controller-7998c67dc4-d9mq2\" (UID: \"c29c7995-8119-4a2d-b7bb-0b233ac8e1c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-d9mq2"
Jul 14 17:37:41.860344 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.860326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c29c7995-8119-4a2d-b7bb-0b233ac8e1c8-cert\") pod \"odh-model-controller-7998c67dc4-d9mq2\" (UID: \"c29c7995-8119-4a2d-b7bb-0b233ac8e1c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-d9mq2"
Jul 14 17:37:41.884364 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:41.884339    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 14 17:37:42.042434 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.042400    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-d9mq2"
Jul 14 17:37:42.221627 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.221601    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 14 17:37:42.225086 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.225008    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-d9mq2"]
Jul 14 17:37:42.227758 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:37:42.227733    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29c7995_8119_4a2d_b7bb_0b233ac8e1c8.slice/crio-368b7c9414df9c8367680dd26f77bd66829c9fa717ee06c91c019eb48739cea8 WatchSource:0}: Error finding container 368b7c9414df9c8367680dd26f77bd66829c9fa717ee06c91c019eb48739cea8: Status 404 returned error can't find the container with id 368b7c9414df9c8367680dd26f77bd66829c9fa717ee06c91c019eb48739cea8
Jul 14 17:37:42.361753 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.361718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd61ccf1-5789-425f-b22b-77db95bd220b-tls-certs\") pod \"kube-auth-proxy-fd75767d5-j5vhv\" (UID: \"cd61ccf1-5789-425f-b22b-77db95bd220b\") " pod="openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv"
Jul 14 17:37:42.364293 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.364263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd61ccf1-5789-425f-b22b-77db95bd220b-tls-certs\") pod \"kube-auth-proxy-fd75767d5-j5vhv\" (UID: \"cd61ccf1-5789-425f-b22b-77db95bd220b\") " pod="openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv"
Jul 14 17:37:42.510128 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.510096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv"
Jul 14 17:37:42.599534 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.599502    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:42.611659 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.611630    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-k497s"]
Jul 14 17:37:42.615238 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.615198    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-k497s"
Jul 14 17:37:42.622703 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.622526    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 14 17:37:42.623065 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.622888    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-pw5px\""
Jul 14 17:37:42.646126 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.646099    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-k497s"]
Jul 14 17:37:42.664878 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.664846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg6hd\" (UniqueName: \"kubernetes.io/projected/d8075eca-f4e2-4595-ab77-3da2f78e0506-kube-api-access-fg6hd\") pod \"model-serving-api-54d89bd5fc-k497s\" (UID: \"d8075eca-f4e2-4595-ab77-3da2f78e0506\") " pod="opendatahub/model-serving-api-54d89bd5fc-k497s"
Jul 14 17:37:42.664998 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.664885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8075eca-f4e2-4595-ab77-3da2f78e0506-tls-certs\") pod \"model-serving-api-54d89bd5fc-k497s\" (UID: \"d8075eca-f4e2-4595-ab77-3da2f78e0506\") " pod="opendatahub/model-serving-api-54d89bd5fc-k497s"
Jul 14 17:37:42.703701 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.703673    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv"]
Jul 14 17:37:42.707620 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:37:42.707595    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd61ccf1_5789_425f_b22b_77db95bd220b.slice/crio-36dd99306e959e23acc25c1dc6913c85dcd458124b31abbd9db666df5eb419b0 WatchSource:0}: Error finding container 36dd99306e959e23acc25c1dc6913c85dcd458124b31abbd9db666df5eb419b0: Status 404 returned error can't find the container with id 36dd99306e959e23acc25c1dc6913c85dcd458124b31abbd9db666df5eb419b0
Jul 14 17:37:42.765720 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.765651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg6hd\" (UniqueName: \"kubernetes.io/projected/d8075eca-f4e2-4595-ab77-3da2f78e0506-kube-api-access-fg6hd\") pod \"model-serving-api-54d89bd5fc-k497s\" (UID: \"d8075eca-f4e2-4595-ab77-3da2f78e0506\") " pod="opendatahub/model-serving-api-54d89bd5fc-k497s"
Jul 14 17:37:42.765720 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.765683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8075eca-f4e2-4595-ab77-3da2f78e0506-tls-certs\") pod \"model-serving-api-54d89bd5fc-k497s\" (UID: \"d8075eca-f4e2-4595-ab77-3da2f78e0506\") " pod="opendatahub/model-serving-api-54d89bd5fc-k497s"
Jul 14 17:37:42.765870 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:37:42.765817    2578 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 14 17:37:42.765870 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:37:42.765868    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8075eca-f4e2-4595-ab77-3da2f78e0506-tls-certs podName:d8075eca-f4e2-4595-ab77-3da2f78e0506 nodeName:}" failed. No retries permitted until 2026-07-14 17:37:43.265852892 +0000 UTC m=+319.982246562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d8075eca-f4e2-4595-ab77-3da2f78e0506-tls-certs") pod "model-serving-api-54d89bd5fc-k497s" (UID: "d8075eca-f4e2-4595-ab77-3da2f78e0506") : secret "model-serving-api-tls" not found
Jul 14 17:37:42.788701 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:42.788672    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg6hd\" (UniqueName: \"kubernetes.io/projected/d8075eca-f4e2-4595-ab77-3da2f78e0506-kube-api-access-fg6hd\") pod \"model-serving-api-54d89bd5fc-k497s\" (UID: \"d8075eca-f4e2-4595-ab77-3da2f78e0506\") " pod="opendatahub/model-serving-api-54d89bd5fc-k497s"
Jul 14 17:37:43.033275 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.033168    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:43.047282 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.047245    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"]
Jul 14 17:37:43.051376 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.051352    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.054692 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.054656    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 14 17:37:43.054692 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.054688    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n8766\""
Jul 14 17:37:43.054893 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.054664    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 14 17:37:43.054893 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.054663    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 17:37:43.055032 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.055012    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 14 17:37:43.070787 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.070739    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"]
Jul 14 17:37:43.103962 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.103922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-d9mq2" event={"ID":"c29c7995-8119-4a2d-b7bb-0b233ac8e1c8","Type":"ContainerStarted","Data":"368b7c9414df9c8367680dd26f77bd66829c9fa717ee06c91c019eb48739cea8"}
Jul 14 17:37:43.105236 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.105195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv" event={"ID":"cd61ccf1-5789-425f-b22b-77db95bd220b","Type":"ContainerStarted","Data":"36dd99306e959e23acc25c1dc6913c85dcd458124b31abbd9db666df5eb419b0"}
Jul 14 17:37:43.141858 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.141828    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:43.169983 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.169948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71014a18-028d-49b7-b310-5e019d0a6676-local-certs\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.170181 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.170008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpxdg\" (UniqueName: \"kubernetes.io/projected/71014a18-028d-49b7-b310-5e019d0a6676-kube-api-access-xpxdg\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.170181 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.170046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.170181 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.170132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71014a18-028d-49b7-b310-5e019d0a6676-istio-token\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.170181 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.170165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.170441 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.170246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-cacerts\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.170441 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.170289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71014a18-028d-49b7-b310-5e019d0a6676-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.270964 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.270929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.271150 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.271001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71014a18-028d-49b7-b310-5e019d0a6676-istio-token\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.271150 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.271028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.271150 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.271088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-cacerts\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.271150 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.271119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8075eca-f4e2-4595-ab77-3da2f78e0506-tls-certs\") pod \"model-serving-api-54d89bd5fc-k497s\" (UID: \"d8075eca-f4e2-4595-ab77-3da2f78e0506\") " pod="opendatahub/model-serving-api-54d89bd5fc-k497s"
Jul 14 17:37:43.271150 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.271142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71014a18-028d-49b7-b310-5e019d0a6676-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.271440 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.271186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71014a18-028d-49b7-b310-5e019d0a6676-local-certs\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.271440 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.271285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxdg\" (UniqueName: \"kubernetes.io/projected/71014a18-028d-49b7-b310-5e019d0a6676-kube-api-access-xpxdg\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.273086 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.273027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71014a18-028d-49b7-b310-5e019d0a6676-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.276952 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.276900    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8075eca-f4e2-4595-ab77-3da2f78e0506-tls-certs\") pod \"model-serving-api-54d89bd5fc-k497s\" (UID: \"d8075eca-f4e2-4595-ab77-3da2f78e0506\") " pod="opendatahub/model-serving-api-54d89bd5fc-k497s"
Jul 14 17:37:43.278322 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.278293    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.278900 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.278876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71014a18-028d-49b7-b310-5e019d0a6676-local-certs\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.279885 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.279862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.280311 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.280291    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-cacerts\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.300540 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.300460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71014a18-028d-49b7-b310-5e019d0a6676-istio-token\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.302951 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.302896    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxdg\" (UniqueName: \"kubernetes.io/projected/71014a18-028d-49b7-b310-5e019d0a6676-kube-api-access-xpxdg\") pod \"istiod-openshift-gateway-7574459656-l72nh\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.349127 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.349073    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:43.363365 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.363330    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:43.530942 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.530902    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-k497s"
Jul 14 17:37:43.548655 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.548314    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"]
Jul 14 17:37:43.551559 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:37:43.551485    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71014a18_028d_49b7_b310_5e019d0a6676.slice/crio-b363ef4845ba8cec9f502e96f4f486f7e86b2139822ca45bdd94a593e6e5dcbb WatchSource:0}: Error finding container b363ef4845ba8cec9f502e96f4f486f7e86b2139822ca45bdd94a593e6e5dcbb: Status 404 returned error can't find the container with id b363ef4845ba8cec9f502e96f4f486f7e86b2139822ca45bdd94a593e6e5dcbb
Jul 14 17:37:43.709486 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:43.709454    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-k497s"]
Jul 14 17:37:44.113176 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:44.113107    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh" event={"ID":"71014a18-028d-49b7-b310-5e019d0a6676","Type":"ContainerStarted","Data":"b363ef4845ba8cec9f502e96f4f486f7e86b2139822ca45bdd94a593e6e5dcbb"}
Jul 14 17:37:44.649980 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:44.649895    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:45.309949 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:37:45.309913    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8075eca_f4e2_4595_ab77_3da2f78e0506.slice/crio-bbfdfec8aedafa867ebeaab820f685d22036c29dc301cef68f4af68df9e12854 WatchSource:0}: Error finding container bbfdfec8aedafa867ebeaab820f685d22036c29dc301cef68f4af68df9e12854: Status 404 returned error can't find the container with id bbfdfec8aedafa867ebeaab820f685d22036c29dc301cef68f4af68df9e12854
Jul 14 17:37:46.124637 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:46.124588    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-k497s" event={"ID":"d8075eca-f4e2-4595-ab77-3da2f78e0506","Type":"ContainerStarted","Data":"bbfdfec8aedafa867ebeaab820f685d22036c29dc301cef68f4af68df9e12854"}
Jul 14 17:37:46.970925 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:46.970877    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 17:37:46.971289 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:46.970971    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 17:37:47.130335 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:47.130296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv" event={"ID":"cd61ccf1-5789-425f-b22b-77db95bd220b","Type":"ContainerStarted","Data":"9c65ab5f28798da55fc1c5aab1c71cc09898492424cb84932a555033c8bf1f6c"}
Jul 14 17:37:47.131788 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:47.131762    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-d9mq2" event={"ID":"c29c7995-8119-4a2d-b7bb-0b233ac8e1c8","Type":"ContainerStarted","Data":"e07eb5be6339b4a68eca80f1801dc1cd5ba9bdb4adcdca94cdeadb4964cf2bdc"}
Jul 14 17:37:47.131932 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:47.131822    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-d9mq2"
Jul 14 17:37:47.133322 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:47.133299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh" event={"ID":"71014a18-028d-49b7-b310-5e019d0a6676","Type":"ContainerStarted","Data":"8b1be473fb1ed667ab079a8b59a8e5ff7cc10eab238a5ed6e8e30bc30bf888cf"}
Jul 14 17:37:47.133420 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:47.133342    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:47.325128 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:47.325022    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-fd75767d5-j5vhv" podStartSLOduration=3.099374404 podStartE2EDuration="7.325005425s" totalImagesPullingTime="4.225631021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:37:47.319708092 +0000 UTC m=+324.036101806" watchObservedRunningTime="2026-07-14 17:37:47.325005425 +0000 UTC m=+324.041399117"
Jul 14 17:37:47.487223 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:47.487147    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh" podStartSLOduration=1.071322019 podStartE2EDuration="4.487129068s" totalImagesPullingTime="3.415807049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:37:47.483766161 +0000 UTC m=+324.200159852" watchObservedRunningTime="2026-07-14 17:37:47.487129068 +0000 UTC m=+324.203522760"
Jul 14 17:37:47.538157 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:47.538097    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-d9mq2" podStartSLOduration=1.802256901 podStartE2EDuration="6.538073782s" totalImagesPullingTime="4.735816881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:37:47.523877428 +0000 UTC m=+324.240271120" watchObservedRunningTime="2026-07-14 17:37:47.538073782 +0000 UTC m=+324.254467475"
Jul 14 17:37:48.140830 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:48.140782    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-l72nh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 17:37:48.141283 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:48.140838    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh" podUID="71014a18-028d-49b7-b310-5e019d0a6676" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 17:37:48.145961 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:48.145916    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:48.256072 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:48.256027    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:48.642174 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:48.642139    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:48.880573 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:48.880501    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:49.020719 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:49.020686    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:49.145273 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:49.145169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-k497s" event={"ID":"d8075eca-f4e2-4595-ab77-3da2f78e0506","Type":"ContainerStarted","Data":"bcd6384f2ce492eaec0fe7612135eaaa675e505094953c77a4d57e45cddc17c1"}
Jul 14 17:37:49.145273 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:49.145268    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-k497s"
Jul 14 17:37:49.309002 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:49.308949    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-k497s" podStartSLOduration=4.156403644 podStartE2EDuration="7.308932861s" totalImagesPullingTime="3.152529217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:37:49.307756657 +0000 UTC m=+326.024150348" watchObservedRunningTime="2026-07-14 17:37:49.308932861 +0000 UTC m=+326.025326553"
Jul 14 17:37:51.139960 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:51.139921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:55.024196 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.024150    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"]
Jul 14 17:37:55.024771 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.024462    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh" podUID="71014a18-028d-49b7-b310-5e019d0a6676" containerName="discovery" containerID="cri-o://8b1be473fb1ed667ab079a8b59a8e5ff7cc10eab238a5ed6e8e30bc30bf888cf" gracePeriod=30
Jul 14 17:37:55.176737 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.176694    2578 generic.go:356] "Generic (PLEG): container finished" podID="71014a18-028d-49b7-b310-5e019d0a6676" containerID="8b1be473fb1ed667ab079a8b59a8e5ff7cc10eab238a5ed6e8e30bc30bf888cf" exitCode=0
Jul 14 17:37:55.176897 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.176757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh" event={"ID":"71014a18-028d-49b7-b310-5e019d0a6676","Type":"ContainerDied","Data":"8b1be473fb1ed667ab079a8b59a8e5ff7cc10eab238a5ed6e8e30bc30bf888cf"}
Jul 14 17:37:55.263805 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.263779    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:55.381287 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.381174    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71014a18-028d-49b7-b310-5e019d0a6676-local-certs\") pod \"71014a18-028d-49b7-b310-5e019d0a6676\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") "
Jul 14 17:37:55.381287 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.381243    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-istio-csr-dns-cert\") pod \"71014a18-028d-49b7-b310-5e019d0a6676\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") "
Jul 14 17:37:55.381287 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.381277    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpxdg\" (UniqueName: \"kubernetes.io/projected/71014a18-028d-49b7-b310-5e019d0a6676-kube-api-access-xpxdg\") pod \"71014a18-028d-49b7-b310-5e019d0a6676\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") "
Jul 14 17:37:55.381572 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.381315    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-istio-kubeconfig\") pod \"71014a18-028d-49b7-b310-5e019d0a6676\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") "
Jul 14 17:37:55.381572 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.381333    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71014a18-028d-49b7-b310-5e019d0a6676-istio-csr-ca-configmap\") pod \"71014a18-028d-49b7-b310-5e019d0a6676\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") "
Jul 14 17:37:55.381572 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.381363    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71014a18-028d-49b7-b310-5e019d0a6676-istio-token\") pod \"71014a18-028d-49b7-b310-5e019d0a6676\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") "
Jul 14 17:37:55.381572 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.381395    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-cacerts\") pod \"71014a18-028d-49b7-b310-5e019d0a6676\" (UID: \"71014a18-028d-49b7-b310-5e019d0a6676\") "
Jul 14 17:37:55.381784 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.381756    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71014a18-028d-49b7-b310-5e019d0a6676-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "71014a18-028d-49b7-b310-5e019d0a6676" (UID: "71014a18-028d-49b7-b310-5e019d0a6676"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:37:55.383828 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.383780    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71014a18-028d-49b7-b310-5e019d0a6676-local-certs" (OuterVolumeSpecName: "local-certs") pod "71014a18-028d-49b7-b310-5e019d0a6676" (UID: "71014a18-028d-49b7-b310-5e019d0a6676"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:37:55.383960 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.383880    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-cacerts" (OuterVolumeSpecName: "cacerts") pod "71014a18-028d-49b7-b310-5e019d0a6676" (UID: "71014a18-028d-49b7-b310-5e019d0a6676"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:37:55.384157 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.384133    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "71014a18-028d-49b7-b310-5e019d0a6676" (UID: "71014a18-028d-49b7-b310-5e019d0a6676"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:37:55.384274 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.384135    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71014a18-028d-49b7-b310-5e019d0a6676-kube-api-access-xpxdg" (OuterVolumeSpecName: "kube-api-access-xpxdg") pod "71014a18-028d-49b7-b310-5e019d0a6676" (UID: "71014a18-028d-49b7-b310-5e019d0a6676"). InnerVolumeSpecName "kube-api-access-xpxdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:37:55.384274 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.384238    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "71014a18-028d-49b7-b310-5e019d0a6676" (UID: "71014a18-028d-49b7-b310-5e019d0a6676"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:37:55.384345 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.384281    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71014a18-028d-49b7-b310-5e019d0a6676-istio-token" (OuterVolumeSpecName: "istio-token") pod "71014a18-028d-49b7-b310-5e019d0a6676" (UID: "71014a18-028d-49b7-b310-5e019d0a6676"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:37:55.482863 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.482820    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-istio-csr-dns-cert\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:37:55.482863 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.482854    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpxdg\" (UniqueName: \"kubernetes.io/projected/71014a18-028d-49b7-b310-5e019d0a6676-kube-api-access-xpxdg\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:37:55.482863 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.482865    2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-istio-kubeconfig\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:37:55.482863 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.482875    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71014a18-028d-49b7-b310-5e019d0a6676-istio-csr-ca-configmap\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:37:55.483138 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.482884    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71014a18-028d-49b7-b310-5e019d0a6676-istio-token\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:37:55.483138 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.482899    2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71014a18-028d-49b7-b310-5e019d0a6676-cacerts\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:37:55.483138 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.482908    2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71014a18-028d-49b7-b310-5e019d0a6676-local-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:37:55.746392 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.746356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:55.835026 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.834990    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr"]
Jul 14 17:37:55.835343 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.835322    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71014a18-028d-49b7-b310-5e019d0a6676" containerName="discovery"
Jul 14 17:37:55.835343 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.835337    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="71014a18-028d-49b7-b310-5e019d0a6676" containerName="discovery"
Jul 14 17:37:55.835480 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.835385    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="71014a18-028d-49b7-b310-5e019d0a6676" containerName="discovery"
Jul 14 17:37:55.837414 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.837396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr"
Jul 14 17:37:55.845170 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.845139    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 14 17:37:55.845170 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.845139    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-cz56x\""
Jul 14 17:37:55.871874 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.871843    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr"]
Jul 14 17:37:55.895762 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.895735    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7"]
Jul 14 17:37:55.898040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.898025    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7"
Jul 14 17:37:55.899890 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.899860    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:55.901498 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.901482    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 14 17:37:55.901673 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.901651    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-z57ss\""
Jul 14 17:37:55.933538 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.933511    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7"]
Jul 14 17:37:55.987430 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.987401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b13f954-fcce-4664-bc31-d1b20cfc25f9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bwbg7\" (UID: \"9b13f954-fcce-4664-bc31-d1b20cfc25f9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7"
Jul 14 17:37:55.987609 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.987445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2ksv\" (UniqueName: \"kubernetes.io/projected/9b13f954-fcce-4664-bc31-d1b20cfc25f9-kube-api-access-w2ksv\") pod \"llmisvc-controller-manager-777c6bd7d5-bwbg7\" (UID: \"9b13f954-fcce-4664-bc31-d1b20cfc25f9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7"
Jul 14 17:37:55.987609 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.987550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f27vd\" (UniqueName: \"kubernetes.io/projected/ef5a476a-8d1d-446f-911d-470497232b26-kube-api-access-f27vd\") pod \"kserve-controller-manager-d6f4b65b8-kr7sr\" (UID: \"ef5a476a-8d1d-446f-911d-470497232b26\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr"
Jul 14 17:37:55.987609 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:55.987592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef5a476a-8d1d-446f-911d-470497232b26-cert\") pod \"kserve-controller-manager-d6f4b65b8-kr7sr\" (UID: \"ef5a476a-8d1d-446f-911d-470497232b26\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr"
Jul 14 17:37:56.004877 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.004802    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:56.088218 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.088171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27vd\" (UniqueName: \"kubernetes.io/projected/ef5a476a-8d1d-446f-911d-470497232b26-kube-api-access-f27vd\") pod \"kserve-controller-manager-d6f4b65b8-kr7sr\" (UID: \"ef5a476a-8d1d-446f-911d-470497232b26\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr"
Jul 14 17:37:56.088666 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.088252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef5a476a-8d1d-446f-911d-470497232b26-cert\") pod \"kserve-controller-manager-d6f4b65b8-kr7sr\" (UID: \"ef5a476a-8d1d-446f-911d-470497232b26\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr"
Jul 14 17:37:56.088666 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.088290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b13f954-fcce-4664-bc31-d1b20cfc25f9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bwbg7\" (UID: \"9b13f954-fcce-4664-bc31-d1b20cfc25f9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7"
Jul 14 17:37:56.088666 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.088334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2ksv\" (UniqueName: \"kubernetes.io/projected/9b13f954-fcce-4664-bc31-d1b20cfc25f9-kube-api-access-w2ksv\") pod \"llmisvc-controller-manager-777c6bd7d5-bwbg7\" (UID: \"9b13f954-fcce-4664-bc31-d1b20cfc25f9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7"
Jul 14 17:37:56.091027 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.091001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b13f954-fcce-4664-bc31-d1b20cfc25f9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bwbg7\" (UID: \"9b13f954-fcce-4664-bc31-d1b20cfc25f9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7"
Jul 14 17:37:56.091142 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.091066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef5a476a-8d1d-446f-911d-470497232b26-cert\") pod \"kserve-controller-manager-d6f4b65b8-kr7sr\" (UID: \"ef5a476a-8d1d-446f-911d-470497232b26\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr"
Jul 14 17:37:56.110877 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.110842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f27vd\" (UniqueName: \"kubernetes.io/projected/ef5a476a-8d1d-446f-911d-470497232b26-kube-api-access-f27vd\") pod \"kserve-controller-manager-d6f4b65b8-kr7sr\" (UID: \"ef5a476a-8d1d-446f-911d-470497232b26\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr"
Jul 14 17:37:56.111017 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.110956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2ksv\" (UniqueName: \"kubernetes.io/projected/9b13f954-fcce-4664-bc31-d1b20cfc25f9-kube-api-access-w2ksv\") pod \"llmisvc-controller-manager-777c6bd7d5-bwbg7\" (UID: \"9b13f954-fcce-4664-bc31-d1b20cfc25f9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7"
Jul 14 17:37:56.148753 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.148722    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr"
Jul 14 17:37:56.185558 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.185512    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh" event={"ID":"71014a18-028d-49b7-b310-5e019d0a6676","Type":"ContainerDied","Data":"b363ef4845ba8cec9f502e96f4f486f7e86b2139822ca45bdd94a593e6e5dcbb"}
Jul 14 17:37:56.185558 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.185547    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"
Jul 14 17:37:56.185807 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.185591    2578 scope.go:117] "RemoveContainer" containerID="8b1be473fb1ed667ab079a8b59a8e5ff7cc10eab238a5ed6e8e30bc30bf888cf"
Jul 14 17:37:56.207820 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.207641    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7"
Jul 14 17:37:56.233180 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.233149    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"]
Jul 14 17:37:56.239557 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.236636    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l72nh"]
Jul 14 17:37:56.370823 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.370793    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr"]
Jul 14 17:37:56.373338 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:37:56.372747    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5a476a_8d1d_446f_911d_470497232b26.slice/crio-f0f345eab9115522f1d37aa4c2acc6e087e00b8968da7d07ce933a741430c5cb WatchSource:0}: Error finding container f0f345eab9115522f1d37aa4c2acc6e087e00b8968da7d07ce933a741430c5cb: Status 404 returned error can't find the container with id f0f345eab9115522f1d37aa4c2acc6e087e00b8968da7d07ce933a741430c5cb
Jul 14 17:37:56.414676 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:56.414642    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7"]
Jul 14 17:37:56.418987 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:37:56.418960    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9b13f954_fcce_4664_bc31_d1b20cfc25f9.slice/crio-3bb2a36bcb20db72dff5e518b35ab06bb6a845bb09326da31a6fe36c3d41388c WatchSource:0}: Error finding container 3bb2a36bcb20db72dff5e518b35ab06bb6a845bb09326da31a6fe36c3d41388c: Status 404 returned error can't find the container with id 3bb2a36bcb20db72dff5e518b35ab06bb6a845bb09326da31a6fe36c3d41388c
Jul 14 17:37:57.159329 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:57.159285    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:57.192308 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:57.192268    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7" event={"ID":"9b13f954-fcce-4664-bc31-d1b20cfc25f9","Type":"ContainerStarted","Data":"3bb2a36bcb20db72dff5e518b35ab06bb6a845bb09326da31a6fe36c3d41388c"}
Jul 14 17:37:57.193922 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:57.193893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr" event={"ID":"ef5a476a-8d1d-446f-911d-470497232b26","Type":"ContainerStarted","Data":"f0f345eab9115522f1d37aa4c2acc6e087e00b8968da7d07ce933a741430c5cb"}
Jul 14 17:37:57.257534 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:57.257497    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:57.355155 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:57.355108    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:37:57.781922 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:57.781892    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71014a18-028d-49b7-b310-5e019d0a6676" path="/var/lib/kubelet/pods/71014a18-028d-49b7-b310-5e019d0a6676/volumes"
Jul 14 17:37:58.142027 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:58.141894    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-d9mq2"
Jul 14 17:37:59.214846 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:59.214803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr" event={"ID":"ef5a476a-8d1d-446f-911d-470497232b26","Type":"ContainerStarted","Data":"32db2ac1f8c957c83bedd6cc72fc8d15ce7b200a9bcb24ad594dc59a0b7a29e4"}
Jul 14 17:37:59.215317 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:59.214990    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr"
Jul 14 17:37:59.243178 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:37:59.243111    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr" podStartSLOduration=1.80298341 podStartE2EDuration="4.243090431s" totalImagesPullingTime="2.440107021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:37:59.24297416 +0000 UTC m=+335.959367854" watchObservedRunningTime="2026-07-14 17:37:59.243090431 +0000 UTC m=+335.959484123"
Jul 14 17:38:00.154045 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:00.154018    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-k497s"
Jul 14 17:38:02.230936 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:02.230891    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7" event={"ID":"9b13f954-fcce-4664-bc31-d1b20cfc25f9","Type":"ContainerStarted","Data":"4cf0c0405e36db7747dfc51d77d352ede0245aa7e7a17b223d4720cf2e01cd16"}
Jul 14 17:38:02.231337 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:02.230985    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7"
Jul 14 17:38:11.869600 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:11.869566    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:38:11.971946 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:11.971915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:38:14.649393 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:14.649312    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:38:14.747562 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:14.747530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:38:30.225043 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:30.225008    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kr7sr"
Jul 14 17:38:30.247143 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:30.247078    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7" podStartSLOduration=30.134686972 podStartE2EDuration="35.247056248s" totalImagesPullingTime="5.112369276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:38:02.256827503 +0000 UTC m=+338.973221194" watchObservedRunningTime="2026-07-14 17:38:30.247056248 +0000 UTC m=+366.963449941"
Jul 14 17:38:33.238115 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:33.238084    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwbg7"
Jul 14 17:38:41.306968 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:41.306931    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:38:41.414660 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:41.414625    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:38:41.483999 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:41.483966    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gkx8w"]
Jul 14 17:38:41.492829 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:41.492803    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gkx8w"
Jul 14 17:38:41.495591 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:41.495563    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 14 17:38:41.495706 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:41.495606    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-qb9hc\""
Jul 14 17:38:41.496534 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:41.496510    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 14 17:38:41.514588 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:41.514557    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gkx8w"]
Jul 14 17:38:41.572463 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:41.572379    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:38:41.581539 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:41.581511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq8n7\" (UniqueName: \"kubernetes.io/projected/415ba6cc-f4cb-4d2c-af6b-f59ee532995e-kube-api-access-hq8n7\") pod \"kuadrant-operator-catalog-gkx8w\" (UID: \"415ba6cc-f4cb-4d2c-af6b-f59ee532995e\") " pod="kuadrant-system/kuadrant-operator-catalog-gkx8w"
Jul 14 17:38:41.682429 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:41.682397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq8n7\" (UniqueName: \"kubernetes.io/projected/415ba6cc-f4cb-4d2c-af6b-f59ee532995e-kube-api-access-hq8n7\") pod \"kuadrant-operator-catalog-gkx8w\" (UID: \"415ba6cc-f4cb-4d2c-af6b-f59ee532995e\") " pod="kuadrant-system/kuadrant-operator-catalog-gkx8w"
Jul 14 17:38:41.698431 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:41.698403    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq8n7\" (UniqueName: \"kubernetes.io/projected/415ba6cc-f4cb-4d2c-af6b-f59ee532995e-kube-api-access-hq8n7\") pod \"kuadrant-operator-catalog-gkx8w\" (UID: \"415ba6cc-f4cb-4d2c-af6b-f59ee532995e\") " pod="kuadrant-system/kuadrant-operator-catalog-gkx8w"
Jul 14 17:38:41.802745 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:41.802706    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gkx8w"]
Jul 14 17:38:41.803011 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:41.802998    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gkx8w"
Jul 14 17:38:41.936577 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:41.936542    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gkx8w"]
Jul 14 17:38:42.018681 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:42.018646    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s425v"]
Jul 14 17:38:42.021489 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:42.021472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-s425v"
Jul 14 17:38:42.034358 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:42.034327    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s425v"]
Jul 14 17:38:42.085349 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:42.085316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4njf\" (UniqueName: \"kubernetes.io/projected/90bc43ea-bf7c-4ffb-88cc-7077753a16b1-kube-api-access-b4njf\") pod \"kuadrant-operator-catalog-s425v\" (UID: \"90bc43ea-bf7c-4ffb-88cc-7077753a16b1\") " pod="kuadrant-system/kuadrant-operator-catalog-s425v"
Jul 14 17:38:42.186319 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:42.186293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4njf\" (UniqueName: \"kubernetes.io/projected/90bc43ea-bf7c-4ffb-88cc-7077753a16b1-kube-api-access-b4njf\") pod \"kuadrant-operator-catalog-s425v\" (UID: \"90bc43ea-bf7c-4ffb-88cc-7077753a16b1\") " pod="kuadrant-system/kuadrant-operator-catalog-s425v"
Jul 14 17:38:42.199013 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:42.198980    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4njf\" (UniqueName: \"kubernetes.io/projected/90bc43ea-bf7c-4ffb-88cc-7077753a16b1-kube-api-access-b4njf\") pod \"kuadrant-operator-catalog-s425v\" (UID: \"90bc43ea-bf7c-4ffb-88cc-7077753a16b1\") " pod="kuadrant-system/kuadrant-operator-catalog-s425v"
Jul 14 17:38:42.330462 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:42.330425    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-s425v"
Jul 14 17:38:42.411900 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:42.411854    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gkx8w" event={"ID":"415ba6cc-f4cb-4d2c-af6b-f59ee532995e","Type":"ContainerStarted","Data":"03197405dc8e36879b020419d84ca8b335cb3c38dae1041ad6f9118059349ac8"}
Jul 14 17:38:42.484895 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:42.484697    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s425v"]
Jul 14 17:38:42.487591 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:38:42.487561    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bc43ea_bf7c_4ffb_88cc_7077753a16b1.slice/crio-a88b6b822c4a248f6f28c2feea83d763f6a68ef3c10c8d25140da1f8847945f6 WatchSource:0}: Error finding container a88b6b822c4a248f6f28c2feea83d763f6a68ef3c10c8d25140da1f8847945f6: Status 404 returned error can't find the container with id a88b6b822c4a248f6f28c2feea83d763f6a68ef3c10c8d25140da1f8847945f6
Jul 14 17:38:43.418558 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:43.418516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-s425v" event={"ID":"90bc43ea-bf7c-4ffb-88cc-7077753a16b1","Type":"ContainerStarted","Data":"a88b6b822c4a248f6f28c2feea83d763f6a68ef3c10c8d25140da1f8847945f6"}
Jul 14 17:38:45.429541 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:45.429509    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gkx8w" event={"ID":"415ba6cc-f4cb-4d2c-af6b-f59ee532995e","Type":"ContainerStarted","Data":"de3327f70c674082b3d2d6fbb4fc750a0be4295758333506c45033a55fdbd0c4"}
Jul 14 17:38:45.429998 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:45.429560    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-gkx8w" podUID="415ba6cc-f4cb-4d2c-af6b-f59ee532995e" containerName="registry-server" containerID="cri-o://de3327f70c674082b3d2d6fbb4fc750a0be4295758333506c45033a55fdbd0c4" gracePeriod=2
Jul 14 17:38:45.431132 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:45.431110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-s425v" event={"ID":"90bc43ea-bf7c-4ffb-88cc-7077753a16b1","Type":"ContainerStarted","Data":"1d6265865f4ff138926ba6bd9fb7c8c44dc882b1f1c99325cd9f576d35edaaef"}
Jul 14 17:38:45.458958 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:45.458912    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-gkx8w" podStartSLOduration=1.802235605 podStartE2EDuration="4.458895691s" totalImagesPullingTime="2.656660086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:38:45.45680618 +0000 UTC m=+382.173199949" watchObservedRunningTime="2026-07-14 17:38:45.458895691 +0000 UTC m=+382.175289383"
Jul 14 17:38:45.668065 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:45.668040    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gkx8w"
Jul 14 17:38:45.708403 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:45.708292    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-s425v" podStartSLOduration=2.599304673 podStartE2EDuration="4.70827025s" totalImagesPullingTime="2.108965577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:38:45.493020426 +0000 UTC m=+382.209414117" watchObservedRunningTime="2026-07-14 17:38:45.70827025 +0000 UTC m=+382.424663943"
Jul 14 17:38:45.817812 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:45.817783    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hq8n7\" (UniqueName: \"kubernetes.io/projected/415ba6cc-f4cb-4d2c-af6b-f59ee532995e-kube-api-access-hq8n7\") pod \"415ba6cc-f4cb-4d2c-af6b-f59ee532995e\" (UID: \"415ba6cc-f4cb-4d2c-af6b-f59ee532995e\") "
Jul 14 17:38:45.820050 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:45.820021    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415ba6cc-f4cb-4d2c-af6b-f59ee532995e-kube-api-access-hq8n7" (OuterVolumeSpecName: "kube-api-access-hq8n7") pod "415ba6cc-f4cb-4d2c-af6b-f59ee532995e" (UID: "415ba6cc-f4cb-4d2c-af6b-f59ee532995e"). InnerVolumeSpecName "kube-api-access-hq8n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:38:45.918911 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:45.918860    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq8n7\" (UniqueName: \"kubernetes.io/projected/415ba6cc-f4cb-4d2c-af6b-f59ee532995e-kube-api-access-hq8n7\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:38:46.402024 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.401990    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bb6466c6d-bfh8h"]
Jul 14 17:38:46.402328 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.402316    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="415ba6cc-f4cb-4d2c-af6b-f59ee532995e" containerName="registry-server"
Jul 14 17:38:46.402385 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.402331    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="415ba6cc-f4cb-4d2c-af6b-f59ee532995e" containerName="registry-server"
Jul 14 17:38:46.402385 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.402383    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="415ba6cc-f4cb-4d2c-af6b-f59ee532995e" containerName="registry-server"
Jul 14 17:38:46.404406 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.404390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.407392 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.407364    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 14 17:38:46.407845 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.407831    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 14 17:38:46.407919 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.407844    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-krthk\""
Jul 14 17:38:46.407919 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.407858    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 14 17:38:46.408404 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.408389    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 14 17:38:46.408476 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.408395    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 14 17:38:46.409324 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.409307    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 14 17:38:46.410146 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.410127    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 14 17:38:46.416098 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.416073    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 14 17:38:46.421594 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.421574    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb6466c6d-bfh8h"]
Jul 14 17:38:46.436407 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.436383    2578 generic.go:356] "Generic (PLEG): container finished" podID="415ba6cc-f4cb-4d2c-af6b-f59ee532995e" containerID="de3327f70c674082b3d2d6fbb4fc750a0be4295758333506c45033a55fdbd0c4" exitCode=0
Jul 14 17:38:46.436711 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.436418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gkx8w" event={"ID":"415ba6cc-f4cb-4d2c-af6b-f59ee532995e","Type":"ContainerDied","Data":"de3327f70c674082b3d2d6fbb4fc750a0be4295758333506c45033a55fdbd0c4"}
Jul 14 17:38:46.436711 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.436439    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gkx8w"
Jul 14 17:38:46.436711 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.436474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gkx8w" event={"ID":"415ba6cc-f4cb-4d2c-af6b-f59ee532995e","Type":"ContainerDied","Data":"03197405dc8e36879b020419d84ca8b335cb3c38dae1041ad6f9118059349ac8"}
Jul 14 17:38:46.436711 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.436497    2578 scope.go:117] "RemoveContainer" containerID="de3327f70c674082b3d2d6fbb4fc750a0be4295758333506c45033a55fdbd0c4"
Jul 14 17:38:46.445630 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.445616    2578 scope.go:117] "RemoveContainer" containerID="de3327f70c674082b3d2d6fbb4fc750a0be4295758333506c45033a55fdbd0c4"
Jul 14 17:38:46.445893 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:38:46.445875    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3327f70c674082b3d2d6fbb4fc750a0be4295758333506c45033a55fdbd0c4\": container with ID starting with de3327f70c674082b3d2d6fbb4fc750a0be4295758333506c45033a55fdbd0c4 not found: ID does not exist" containerID="de3327f70c674082b3d2d6fbb4fc750a0be4295758333506c45033a55fdbd0c4"
Jul 14 17:38:46.445937 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.445901    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3327f70c674082b3d2d6fbb4fc750a0be4295758333506c45033a55fdbd0c4"} err="failed to get container status \"de3327f70c674082b3d2d6fbb4fc750a0be4295758333506c45033a55fdbd0c4\": rpc error: code = NotFound desc = could not find container \"de3327f70c674082b3d2d6fbb4fc750a0be4295758333506c45033a55fdbd0c4\": container with ID starting with de3327f70c674082b3d2d6fbb4fc750a0be4295758333506c45033a55fdbd0c4 not found: ID does not exist"
Jul 14 17:38:46.462817 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.462796    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gkx8w"]
Jul 14 17:38:46.473330 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.473311    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gkx8w"]
Jul 14 17:38:46.476019 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.475998    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:38:46.524003 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.523976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-console-oauth-config\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.524122 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.524017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-console-serving-cert\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.524122 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.524052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-service-ca\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.524122 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.524077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-oauth-serving-cert\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.524299 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.524281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b4qr\" (UniqueName: \"kubernetes.io/projected/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-kube-api-access-4b4qr\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.524346 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.524309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-trusted-ca-bundle\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.524346 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.524332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-console-config\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.560022 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.559994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:38:46.625479 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.625441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-console-oauth-config\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.625649 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.625503    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-console-serving-cert\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.625649 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.625537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-service-ca\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.625649 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.625558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-oauth-serving-cert\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.625649 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.625588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b4qr\" (UniqueName: \"kubernetes.io/projected/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-kube-api-access-4b4qr\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.625649 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.625613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-trusted-ca-bundle\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.625649 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.625636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-console-config\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.626395 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.626374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-console-config\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.626571 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.626539    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-service-ca\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.626627 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.626539    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-oauth-serving-cert\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.626835 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.626813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-trusted-ca-bundle\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.628025 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.628004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-console-oauth-config\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.628220 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.628186    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-console-serving-cert\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.638579 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.638549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b4qr\" (UniqueName: \"kubernetes.io/projected/f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d-kube-api-access-4b4qr\") pod \"console-bb6466c6d-bfh8h\" (UID: \"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d\") " pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.703688 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.703654    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:38:46.717625 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.717600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:46.854454 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:46.854375    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb6466c6d-bfh8h"]
Jul 14 17:38:46.856505 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:38:46.856477    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a5b360_82f8_4b9c_8ce2_1b1516b82f2d.slice/crio-5004255a6458da4ecfb94b05c667e320727ab6ae42682c7633d658194894e8f8 WatchSource:0}: Error finding container 5004255a6458da4ecfb94b05c667e320727ab6ae42682c7633d658194894e8f8: Status 404 returned error can't find the container with id 5004255a6458da4ecfb94b05c667e320727ab6ae42682c7633d658194894e8f8
Jul 14 17:38:47.275222 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:47.275159    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:38:47.442430 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:47.442394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb6466c6d-bfh8h" event={"ID":"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d","Type":"ContainerStarted","Data":"5004255a6458da4ecfb94b05c667e320727ab6ae42682c7633d658194894e8f8"}
Jul 14 17:38:47.451739 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:47.451712    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:38:47.783749 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:47.783714    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="415ba6cc-f4cb-4d2c-af6b-f59ee532995e" path="/var/lib/kubelet/pods/415ba6cc-f4cb-4d2c-af6b-f59ee532995e/volumes"
Jul 14 17:38:50.460555 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:50.460520    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb6466c6d-bfh8h" event={"ID":"f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d","Type":"ContainerStarted","Data":"42cfaa161e9ff68920848a5a2aaf4357a901167bfef724dd837c951e98222eff"}
Jul 14 17:38:50.497370 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:50.497316    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bb6466c6d-bfh8h" podStartSLOduration=1.36121044 podStartE2EDuration="4.49730026s" totalImagesPullingTime="3.13608982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:38:50.496727627 +0000 UTC m=+387.213121319" watchObservedRunningTime="2026-07-14 17:38:50.49730026 +0000 UTC m=+387.213693953"
Jul 14 17:38:52.331047 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:52.331008    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-s425v"
Jul 14 17:38:52.331462 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:52.331071    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-s425v"
Jul 14 17:38:52.353832 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:52.353807    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-s425v"
Jul 14 17:38:52.492261 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:52.492234    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-s425v"
Jul 14 17:38:56.717988 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:56.717944    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:56.718455 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:56.718027    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:56.723038 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:56.723016    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:57.123658 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.123566    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"]
Jul 14 17:38:57.126260 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.126238    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"
Jul 14 17:38:57.129512 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.129494    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ms5zj\""
Jul 14 17:38:57.150255 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.150227    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"]
Jul 14 17:38:57.210841 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.210806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp\" (UID: \"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"
Jul 14 17:38:57.211027 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.210858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp\" (UID: \"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"
Jul 14 17:38:57.211027 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.210945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p8zq\" (UniqueName: \"kubernetes.io/projected/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-kube-api-access-7p8zq\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp\" (UID: \"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"
Jul 14 17:38:57.311573 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.311539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp\" (UID: \"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"
Jul 14 17:38:57.311728 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.311592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp\" (UID: \"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"
Jul 14 17:38:57.311728 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.311632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p8zq\" (UniqueName: \"kubernetes.io/projected/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-kube-api-access-7p8zq\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp\" (UID: \"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"
Jul 14 17:38:57.311934 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.311915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp\" (UID: \"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"
Jul 14 17:38:57.312007 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.311991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp\" (UID: \"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"
Jul 14 17:38:57.327917 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.327886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p8zq\" (UniqueName: \"kubernetes.io/projected/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-kube-api-access-7p8zq\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp\" (UID: \"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"
Jul 14 17:38:57.435784 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.435760    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"
Jul 14 17:38:57.504978 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.504948    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bb6466c6d-bfh8h"
Jul 14 17:38:57.575563 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.575539    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"]
Jul 14 17:38:57.577716 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:38:57.577672    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7df75c1_34d5_4e45_8c2f_d30cc4d29d48.slice/crio-78c5f84d93320deb1609cddf62f36e29823063965e4428e6a69c915a1446cba6 WatchSource:0}: Error finding container 78c5f84d93320deb1609cddf62f36e29823063965e4428e6a69c915a1446cba6: Status 404 returned error can't find the container with id 78c5f84d93320deb1609cddf62f36e29823063965e4428e6a69c915a1446cba6
Jul 14 17:38:57.744159 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.744127    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"]
Jul 14 17:38:57.746807 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.746788    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"
Jul 14 17:38:57.766462 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.766435    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"]
Jul 14 17:38:57.799197 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.799164    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:38:57.882550 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.882517    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:38:57.916971 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.916940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzdmw\" (UniqueName: \"kubernetes.io/projected/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-kube-api-access-dzdmw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp\" (UID: \"24621efc-cea7-4df7-a5af-e02a4ab5a8d2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"
Jul 14 17:38:57.917095 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.916991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp\" (UID: \"24621efc-cea7-4df7-a5af-e02a4ab5a8d2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"
Jul 14 17:38:57.917095 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:57.917084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp\" (UID: \"24621efc-cea7-4df7-a5af-e02a4ab5a8d2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"
Jul 14 17:38:58.018018 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.017927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp\" (UID: \"24621efc-cea7-4df7-a5af-e02a4ab5a8d2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"
Jul 14 17:38:58.018018 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.017983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzdmw\" (UniqueName: \"kubernetes.io/projected/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-kube-api-access-dzdmw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp\" (UID: \"24621efc-cea7-4df7-a5af-e02a4ab5a8d2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"
Jul 14 17:38:58.018018 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.018017    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp\" (UID: \"24621efc-cea7-4df7-a5af-e02a4ab5a8d2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"
Jul 14 17:38:58.018364 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.018343    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp\" (UID: \"24621efc-cea7-4df7-a5af-e02a4ab5a8d2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"
Jul 14 17:38:58.018408 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.018394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp\" (UID: \"24621efc-cea7-4df7-a5af-e02a4ab5a8d2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"
Jul 14 17:38:58.028512 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.028492    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzdmw\" (UniqueName: \"kubernetes.io/projected/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-kube-api-access-dzdmw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp\" (UID: \"24621efc-cea7-4df7-a5af-e02a4ab5a8d2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"
Jul 14 17:38:58.055326 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.055298    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"
Jul 14 17:38:58.192982 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.192958    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"]
Jul 14 17:38:58.195463 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:38:58.195433    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24621efc_cea7_4df7_a5af_e02a4ab5a8d2.slice/crio-e7c4e666c518f5176200484153c7ca1fb5899445178a6ddaf84a54079d9825bb WatchSource:0}: Error finding container e7c4e666c518f5176200484153c7ca1fb5899445178a6ddaf84a54079d9825bb: Status 404 returned error can't find the container with id e7c4e666c518f5176200484153c7ca1fb5899445178a6ddaf84a54079d9825bb
Jul 14 17:38:58.350220 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.350179    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"]
Jul 14 17:38:58.352899 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.352878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"
Jul 14 17:38:58.370193 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.370169    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"]
Jul 14 17:38:58.506671 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.506641    2578 generic.go:356] "Generic (PLEG): container finished" podID="d7df75c1-34d5-4e45-8c2f-d30cc4d29d48" containerID="3476bca7edf3fa65f94c77be00e3978bda24c5211b8a8a9f3e9cf1020eb29a94" exitCode=0
Jul 14 17:38:58.506850 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.506736    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp" event={"ID":"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48","Type":"ContainerDied","Data":"3476bca7edf3fa65f94c77be00e3978bda24c5211b8a8a9f3e9cf1020eb29a94"}
Jul 14 17:38:58.506850 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.506797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp" event={"ID":"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48","Type":"ContainerStarted","Data":"78c5f84d93320deb1609cddf62f36e29823063965e4428e6a69c915a1446cba6"}
Jul 14 17:38:58.508379 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.508358    2578 generic.go:356] "Generic (PLEG): container finished" podID="24621efc-cea7-4df7-a5af-e02a4ab5a8d2" containerID="cd7356a62028fb43feedd214e9020ea8c9a2fb824708714f0733403aea8b4b43" exitCode=0
Jul 14 17:38:58.508504 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.508442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp" event={"ID":"24621efc-cea7-4df7-a5af-e02a4ab5a8d2","Type":"ContainerDied","Data":"cd7356a62028fb43feedd214e9020ea8c9a2fb824708714f0733403aea8b4b43"}
Jul 14 17:38:58.508504 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.508485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp" event={"ID":"24621efc-cea7-4df7-a5af-e02a4ab5a8d2","Type":"ContainerStarted","Data":"e7c4e666c518f5176200484153c7ca1fb5899445178a6ddaf84a54079d9825bb"}
Jul 14 17:38:58.522181 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.522157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b693041-1102-4307-ade2-6afd89206f18-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5\" (UID: \"2b693041-1102-4307-ade2-6afd89206f18\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"
Jul 14 17:38:58.522272 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.522189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b693041-1102-4307-ade2-6afd89206f18-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5\" (UID: \"2b693041-1102-4307-ade2-6afd89206f18\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"
Jul 14 17:38:58.522328 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.522310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9m9n\" (UniqueName: \"kubernetes.io/projected/2b693041-1102-4307-ade2-6afd89206f18-kube-api-access-l9m9n\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5\" (UID: \"2b693041-1102-4307-ade2-6afd89206f18\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"
Jul 14 17:38:58.623164 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.623080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9m9n\" (UniqueName: \"kubernetes.io/projected/2b693041-1102-4307-ade2-6afd89206f18-kube-api-access-l9m9n\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5\" (UID: \"2b693041-1102-4307-ade2-6afd89206f18\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"
Jul 14 17:38:58.623368 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.623164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b693041-1102-4307-ade2-6afd89206f18-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5\" (UID: \"2b693041-1102-4307-ade2-6afd89206f18\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"
Jul 14 17:38:58.623368 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.623241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b693041-1102-4307-ade2-6afd89206f18-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5\" (UID: \"2b693041-1102-4307-ade2-6afd89206f18\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"
Jul 14 17:38:58.623649 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.623624    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b693041-1102-4307-ade2-6afd89206f18-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5\" (UID: \"2b693041-1102-4307-ade2-6afd89206f18\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"
Jul 14 17:38:58.623720 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.623659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b693041-1102-4307-ade2-6afd89206f18-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5\" (UID: \"2b693041-1102-4307-ade2-6afd89206f18\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"
Jul 14 17:38:58.633807 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.633774    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9m9n\" (UniqueName: \"kubernetes.io/projected/2b693041-1102-4307-ade2-6afd89206f18-kube-api-access-l9m9n\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5\" (UID: \"2b693041-1102-4307-ade2-6afd89206f18\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"
Jul 14 17:38:58.662296 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.662265    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"
Jul 14 17:38:58.714560 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.707450    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"]
Jul 14 17:38:58.715961 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.715928    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"
Jul 14 17:38:58.730955 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.730895    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"]
Jul 14 17:38:58.813332 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.813309    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"]
Jul 14 17:38:58.815258 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:38:58.815225    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b693041_1102_4307_ade2_6afd89206f18.slice/crio-a4a6864547b14e4398d9e21ee62d148b98ce45900ebc9dbf7d293de6419668c9 WatchSource:0}: Error finding container a4a6864547b14e4398d9e21ee62d148b98ce45900ebc9dbf7d293de6419668c9: Status 404 returned error can't find the container with id a4a6864547b14e4398d9e21ee62d148b98ce45900ebc9dbf7d293de6419668c9
Jul 14 17:38:58.824552 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.824527    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4be48f74-1675-42e1-a07f-6207e22d2a18-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd\" (UID: \"4be48f74-1675-42e1-a07f-6207e22d2a18\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"
Jul 14 17:38:58.824733 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.824587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9qkh\" (UniqueName: \"kubernetes.io/projected/4be48f74-1675-42e1-a07f-6207e22d2a18-kube-api-access-v9qkh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd\" (UID: \"4be48f74-1675-42e1-a07f-6207e22d2a18\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"
Jul 14 17:38:58.824733 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.824658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4be48f74-1675-42e1-a07f-6207e22d2a18-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd\" (UID: \"4be48f74-1675-42e1-a07f-6207e22d2a18\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"
Jul 14 17:38:58.925284 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.925256    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9qkh\" (UniqueName: \"kubernetes.io/projected/4be48f74-1675-42e1-a07f-6207e22d2a18-kube-api-access-v9qkh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd\" (UID: \"4be48f74-1675-42e1-a07f-6207e22d2a18\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"
Jul 14 17:38:58.925439 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.925317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4be48f74-1675-42e1-a07f-6207e22d2a18-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd\" (UID: \"4be48f74-1675-42e1-a07f-6207e22d2a18\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"
Jul 14 17:38:58.925439 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.925347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4be48f74-1675-42e1-a07f-6207e22d2a18-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd\" (UID: \"4be48f74-1675-42e1-a07f-6207e22d2a18\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"
Jul 14 17:38:58.925732 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.925712    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4be48f74-1675-42e1-a07f-6207e22d2a18-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd\" (UID: \"4be48f74-1675-42e1-a07f-6207e22d2a18\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"
Jul 14 17:38:58.925809 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.925749    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4be48f74-1675-42e1-a07f-6207e22d2a18-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd\" (UID: \"4be48f74-1675-42e1-a07f-6207e22d2a18\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"
Jul 14 17:38:58.939531 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:58.939506    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9qkh\" (UniqueName: \"kubernetes.io/projected/4be48f74-1675-42e1-a07f-6207e22d2a18-kube-api-access-v9qkh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd\" (UID: \"4be48f74-1675-42e1-a07f-6207e22d2a18\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"
Jul 14 17:38:59.029611 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:59.029575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"
Jul 14 17:38:59.176970 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:59.176856    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"]
Jul 14 17:38:59.179427 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:38:59.179400    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be48f74_1675_42e1_a07f_6207e22d2a18.slice/crio-567a95ea95bad6192cb89f07c626cbcd5761d887f090d996e12f46a29e5a4860 WatchSource:0}: Error finding container 567a95ea95bad6192cb89f07c626cbcd5761d887f090d996e12f46a29e5a4860: Status 404 returned error can't find the container with id 567a95ea95bad6192cb89f07c626cbcd5761d887f090d996e12f46a29e5a4860
Jul 14 17:38:59.517670 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:59.517640    2578 generic.go:356] "Generic (PLEG): container finished" podID="24621efc-cea7-4df7-a5af-e02a4ab5a8d2" containerID="4646ed1657aca4e7ecb4b8cc7485e3bc5960564f8b9dc4e8c37112fc4221d88f" exitCode=0
Jul 14 17:38:59.517835 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:59.517717    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp" event={"ID":"24621efc-cea7-4df7-a5af-e02a4ab5a8d2","Type":"ContainerDied","Data":"4646ed1657aca4e7ecb4b8cc7485e3bc5960564f8b9dc4e8c37112fc4221d88f"}
Jul 14 17:38:59.519357 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:59.519329    2578 generic.go:356] "Generic (PLEG): container finished" podID="4be48f74-1675-42e1-a07f-6207e22d2a18" containerID="c7de4cdd64f5d3a1efb9c2db76b775364f16af58190be002435c4d8002e5df90" exitCode=0
Jul 14 17:38:59.519438 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:59.519407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd" event={"ID":"4be48f74-1675-42e1-a07f-6207e22d2a18","Type":"ContainerDied","Data":"c7de4cdd64f5d3a1efb9c2db76b775364f16af58190be002435c4d8002e5df90"}
Jul 14 17:38:59.519504 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:59.519455    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd" event={"ID":"4be48f74-1675-42e1-a07f-6207e22d2a18","Type":"ContainerStarted","Data":"567a95ea95bad6192cb89f07c626cbcd5761d887f090d996e12f46a29e5a4860"}
Jul 14 17:38:59.520928 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:59.520908    2578 generic.go:356] "Generic (PLEG): container finished" podID="2b693041-1102-4307-ade2-6afd89206f18" containerID="955b27df92bcefbea3eef32d8050a9bbe82a52481b04ccd8b2e724e8d95be63f" exitCode=0
Jul 14 17:38:59.521042 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:59.520953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5" event={"ID":"2b693041-1102-4307-ade2-6afd89206f18","Type":"ContainerDied","Data":"955b27df92bcefbea3eef32d8050a9bbe82a52481b04ccd8b2e724e8d95be63f"}
Jul 14 17:38:59.521042 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:38:59.520984    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5" event={"ID":"2b693041-1102-4307-ade2-6afd89206f18","Type":"ContainerStarted","Data":"a4a6864547b14e4398d9e21ee62d148b98ce45900ebc9dbf7d293de6419668c9"}
Jul 14 17:39:00.528155 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:00.528126    2578 generic.go:356] "Generic (PLEG): container finished" podID="24621efc-cea7-4df7-a5af-e02a4ab5a8d2" containerID="636a981935a5d77b803c7387073145e0a92d3fbce8e9696db8d1a40a93ba8222" exitCode=0
Jul 14 17:39:00.528578 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:00.528228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp" event={"ID":"24621efc-cea7-4df7-a5af-e02a4ab5a8d2","Type":"ContainerDied","Data":"636a981935a5d77b803c7387073145e0a92d3fbce8e9696db8d1a40a93ba8222"}
Jul 14 17:39:00.530047 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:00.530022    2578 generic.go:356] "Generic (PLEG): container finished" podID="2b693041-1102-4307-ade2-6afd89206f18" containerID="ceb35f538c9597fbb85b9ee082efdc4dd87063614047d4fdf7bb8f8fea80006d" exitCode=0
Jul 14 17:39:00.530136 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:00.530087    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5" event={"ID":"2b693041-1102-4307-ade2-6afd89206f18","Type":"ContainerDied","Data":"ceb35f538c9597fbb85b9ee082efdc4dd87063614047d4fdf7bb8f8fea80006d"}
Jul 14 17:39:01.537560 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:01.537528    2578 generic.go:356] "Generic (PLEG): container finished" podID="4be48f74-1675-42e1-a07f-6207e22d2a18" containerID="113440ac93efbfc9024b085cfc2080d4dd141fc4972ca43c6637a25544349727" exitCode=0
Jul 14 17:39:01.537963 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:01.537621    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd" event={"ID":"4be48f74-1675-42e1-a07f-6207e22d2a18","Type":"ContainerDied","Data":"113440ac93efbfc9024b085cfc2080d4dd141fc4972ca43c6637a25544349727"}
Jul 14 17:39:01.539905 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:01.539874    2578 generic.go:356] "Generic (PLEG): container finished" podID="2b693041-1102-4307-ade2-6afd89206f18" containerID="5b9569b3c04727dec0178af077fc423e5238e9156835566083a778bfa8dd913d" exitCode=0
Jul 14 17:39:01.540024 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:01.539944    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5" event={"ID":"2b693041-1102-4307-ade2-6afd89206f18","Type":"ContainerDied","Data":"5b9569b3c04727dec0178af077fc423e5238e9156835566083a778bfa8dd913d"}
Jul 14 17:39:01.788522 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:01.788457    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"
Jul 14 17:39:01.947874 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:01.947841    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-util\") pod \"24621efc-cea7-4df7-a5af-e02a4ab5a8d2\" (UID: \"24621efc-cea7-4df7-a5af-e02a4ab5a8d2\") "
Jul 14 17:39:01.948040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:01.947886    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-bundle\") pod \"24621efc-cea7-4df7-a5af-e02a4ab5a8d2\" (UID: \"24621efc-cea7-4df7-a5af-e02a4ab5a8d2\") "
Jul 14 17:39:01.948040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:01.947922    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzdmw\" (UniqueName: \"kubernetes.io/projected/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-kube-api-access-dzdmw\") pod \"24621efc-cea7-4df7-a5af-e02a4ab5a8d2\" (UID: \"24621efc-cea7-4df7-a5af-e02a4ab5a8d2\") "
Jul 14 17:39:01.948636 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:01.948604    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-bundle" (OuterVolumeSpecName: "bundle") pod "24621efc-cea7-4df7-a5af-e02a4ab5a8d2" (UID: "24621efc-cea7-4df7-a5af-e02a4ab5a8d2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:39:01.950111 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:01.950080    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-kube-api-access-dzdmw" (OuterVolumeSpecName: "kube-api-access-dzdmw") pod "24621efc-cea7-4df7-a5af-e02a4ab5a8d2" (UID: "24621efc-cea7-4df7-a5af-e02a4ab5a8d2"). InnerVolumeSpecName "kube-api-access-dzdmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:39:01.953621 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:01.953600    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-util" (OuterVolumeSpecName: "util") pod "24621efc-cea7-4df7-a5af-e02a4ab5a8d2" (UID: "24621efc-cea7-4df7-a5af-e02a4ab5a8d2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:39:02.049234 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.049130    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-util\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:39:02.049234 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.049157    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-bundle\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:39:02.049234 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.049169    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzdmw\" (UniqueName: \"kubernetes.io/projected/24621efc-cea7-4df7-a5af-e02a4ab5a8d2-kube-api-access-dzdmw\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:39:02.546936 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.546898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp" event={"ID":"24621efc-cea7-4df7-a5af-e02a4ab5a8d2","Type":"ContainerDied","Data":"e7c4e666c518f5176200484153c7ca1fb5899445178a6ddaf84a54079d9825bb"}
Jul 14 17:39:02.547469 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.546958    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7c4e666c518f5176200484153c7ca1fb5899445178a6ddaf84a54079d9825bb"
Jul 14 17:39:02.547469 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.546913    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp"
Jul 14 17:39:02.549304 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.549279    2578 generic.go:356] "Generic (PLEG): container finished" podID="4be48f74-1675-42e1-a07f-6207e22d2a18" containerID="57ce16e3e0aa7c07d776e3d4d61ef5e27324b8500acf2900c75a9a6a0e87d478" exitCode=0
Jul 14 17:39:02.549414 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.549351    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd" event={"ID":"4be48f74-1675-42e1-a07f-6207e22d2a18","Type":"ContainerDied","Data":"57ce16e3e0aa7c07d776e3d4d61ef5e27324b8500acf2900c75a9a6a0e87d478"}
Jul 14 17:39:02.551152 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.551126    2578 generic.go:356] "Generic (PLEG): container finished" podID="d7df75c1-34d5-4e45-8c2f-d30cc4d29d48" containerID="42d0cfd81dfae8eab8f7a4d1f51cd5b8de3c0781ce7613088fc6842ac0892b1e" exitCode=0
Jul 14 17:39:02.551264 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.551168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp" event={"ID":"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48","Type":"ContainerDied","Data":"42d0cfd81dfae8eab8f7a4d1f51cd5b8de3c0781ce7613088fc6842ac0892b1e"}
Jul 14 17:39:02.707927 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.707904    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"
Jul 14 17:39:02.856194 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.856105    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9m9n\" (UniqueName: \"kubernetes.io/projected/2b693041-1102-4307-ade2-6afd89206f18-kube-api-access-l9m9n\") pod \"2b693041-1102-4307-ade2-6afd89206f18\" (UID: \"2b693041-1102-4307-ade2-6afd89206f18\") "
Jul 14 17:39:02.856194 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.856160    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b693041-1102-4307-ade2-6afd89206f18-util\") pod \"2b693041-1102-4307-ade2-6afd89206f18\" (UID: \"2b693041-1102-4307-ade2-6afd89206f18\") "
Jul 14 17:39:02.856194 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.856191    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b693041-1102-4307-ade2-6afd89206f18-bundle\") pod \"2b693041-1102-4307-ade2-6afd89206f18\" (UID: \"2b693041-1102-4307-ade2-6afd89206f18\") "
Jul 14 17:39:02.856714 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.856688    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b693041-1102-4307-ade2-6afd89206f18-bundle" (OuterVolumeSpecName: "bundle") pod "2b693041-1102-4307-ade2-6afd89206f18" (UID: "2b693041-1102-4307-ade2-6afd89206f18"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:39:02.858611 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.858584    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b693041-1102-4307-ade2-6afd89206f18-kube-api-access-l9m9n" (OuterVolumeSpecName: "kube-api-access-l9m9n") pod "2b693041-1102-4307-ade2-6afd89206f18" (UID: "2b693041-1102-4307-ade2-6afd89206f18"). InnerVolumeSpecName "kube-api-access-l9m9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:39:02.863436 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.863334    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b693041-1102-4307-ade2-6afd89206f18-util" (OuterVolumeSpecName: "util") pod "2b693041-1102-4307-ade2-6afd89206f18" (UID: "2b693041-1102-4307-ade2-6afd89206f18"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:39:02.957716 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.957682    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9m9n\" (UniqueName: \"kubernetes.io/projected/2b693041-1102-4307-ade2-6afd89206f18-kube-api-access-l9m9n\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:39:02.957716 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.957710    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b693041-1102-4307-ade2-6afd89206f18-util\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:39:02.957716 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:02.957721    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b693041-1102-4307-ade2-6afd89206f18-bundle\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:39:03.558524 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:03.558485    2578 generic.go:356] "Generic (PLEG): container finished" podID="d7df75c1-34d5-4e45-8c2f-d30cc4d29d48" containerID="31b6da4cf60921bd7b2c783f65b56d9243c7b2ab73a0725746e7e5a3e2afa43c" exitCode=0
Jul 14 17:39:03.558961 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:03.558557    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp" event={"ID":"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48","Type":"ContainerDied","Data":"31b6da4cf60921bd7b2c783f65b56d9243c7b2ab73a0725746e7e5a3e2afa43c"}
Jul 14 17:39:03.560391 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:03.560368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5" event={"ID":"2b693041-1102-4307-ade2-6afd89206f18","Type":"ContainerDied","Data":"a4a6864547b14e4398d9e21ee62d148b98ce45900ebc9dbf7d293de6419668c9"}
Jul 14 17:39:03.560502 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:03.560404    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5"
Jul 14 17:39:03.560502 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:03.560408    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4a6864547b14e4398d9e21ee62d148b98ce45900ebc9dbf7d293de6419668c9"
Jul 14 17:39:03.691657 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:03.691631    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"
Jul 14 17:39:03.866375 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:03.866285    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4be48f74-1675-42e1-a07f-6207e22d2a18-bundle\") pod \"4be48f74-1675-42e1-a07f-6207e22d2a18\" (UID: \"4be48f74-1675-42e1-a07f-6207e22d2a18\") "
Jul 14 17:39:03.866375 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:03.866326    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9qkh\" (UniqueName: \"kubernetes.io/projected/4be48f74-1675-42e1-a07f-6207e22d2a18-kube-api-access-v9qkh\") pod \"4be48f74-1675-42e1-a07f-6207e22d2a18\" (UID: \"4be48f74-1675-42e1-a07f-6207e22d2a18\") "
Jul 14 17:39:03.866375 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:03.866350    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4be48f74-1675-42e1-a07f-6207e22d2a18-util\") pod \"4be48f74-1675-42e1-a07f-6207e22d2a18\" (UID: \"4be48f74-1675-42e1-a07f-6207e22d2a18\") "
Jul 14 17:39:03.866903 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:03.866872    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be48f74-1675-42e1-a07f-6207e22d2a18-bundle" (OuterVolumeSpecName: "bundle") pod "4be48f74-1675-42e1-a07f-6207e22d2a18" (UID: "4be48f74-1675-42e1-a07f-6207e22d2a18"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:39:03.868614 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:03.868593    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be48f74-1675-42e1-a07f-6207e22d2a18-kube-api-access-v9qkh" (OuterVolumeSpecName: "kube-api-access-v9qkh") pod "4be48f74-1675-42e1-a07f-6207e22d2a18" (UID: "4be48f74-1675-42e1-a07f-6207e22d2a18"). InnerVolumeSpecName "kube-api-access-v9qkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:39:03.871810 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:03.871773    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be48f74-1675-42e1-a07f-6207e22d2a18-util" (OuterVolumeSpecName: "util") pod "4be48f74-1675-42e1-a07f-6207e22d2a18" (UID: "4be48f74-1675-42e1-a07f-6207e22d2a18"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:39:03.967880 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:03.967829    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4be48f74-1675-42e1-a07f-6207e22d2a18-bundle\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:39:03.967880 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:03.967876    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9qkh\" (UniqueName: \"kubernetes.io/projected/4be48f74-1675-42e1-a07f-6207e22d2a18-kube-api-access-v9qkh\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:39:03.967880 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:03.967888    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4be48f74-1675-42e1-a07f-6207e22d2a18-util\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:39:04.567637 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:04.567603    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd"
Jul 14 17:39:04.567637 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:04.567625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd" event={"ID":"4be48f74-1675-42e1-a07f-6207e22d2a18","Type":"ContainerDied","Data":"567a95ea95bad6192cb89f07c626cbcd5761d887f090d996e12f46a29e5a4860"}
Jul 14 17:39:04.568031 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:04.567679    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="567a95ea95bad6192cb89f07c626cbcd5761d887f090d996e12f46a29e5a4860"
Jul 14 17:39:04.692453 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:04.692429    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"
Jul 14 17:39:04.876308 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:04.876198    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-util\") pod \"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48\" (UID: \"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48\") "
Jul 14 17:39:04.876308 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:04.876273    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-bundle\") pod \"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48\" (UID: \"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48\") "
Jul 14 17:39:04.876539 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:04.876366    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p8zq\" (UniqueName: \"kubernetes.io/projected/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-kube-api-access-7p8zq\") pod \"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48\" (UID: \"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48\") "
Jul 14 17:39:04.876780 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:04.876757    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-bundle" (OuterVolumeSpecName: "bundle") pod "d7df75c1-34d5-4e45-8c2f-d30cc4d29d48" (UID: "d7df75c1-34d5-4e45-8c2f-d30cc4d29d48"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:39:04.878526 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:04.878489    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-kube-api-access-7p8zq" (OuterVolumeSpecName: "kube-api-access-7p8zq") pod "d7df75c1-34d5-4e45-8c2f-d30cc4d29d48" (UID: "d7df75c1-34d5-4e45-8c2f-d30cc4d29d48"). InnerVolumeSpecName "kube-api-access-7p8zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:39:04.881058 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:04.881030    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-util" (OuterVolumeSpecName: "util") pod "d7df75c1-34d5-4e45-8c2f-d30cc4d29d48" (UID: "d7df75c1-34d5-4e45-8c2f-d30cc4d29d48"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:39:04.977930 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:04.977894    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p8zq\" (UniqueName: \"kubernetes.io/projected/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-kube-api-access-7p8zq\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:39:04.977930 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:04.977927    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-util\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:39:04.977930 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:04.977940    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7df75c1-34d5-4e45-8c2f-d30cc4d29d48-bundle\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:39:05.574257 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:05.574223    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp" event={"ID":"d7df75c1-34d5-4e45-8c2f-d30cc4d29d48","Type":"ContainerDied","Data":"78c5f84d93320deb1609cddf62f36e29823063965e4428e6a69c915a1446cba6"}
Jul 14 17:39:05.574625 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:05.574267    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp"
Jul 14 17:39:05.574625 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:05.574270    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78c5f84d93320deb1609cddf62f36e29823063965e4428e6a69c915a1446cba6"
Jul 14 17:39:10.894969 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:10.894932    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:13.585485 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585447    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"]
Jul 14 17:39:13.585842 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585789    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be48f74-1675-42e1-a07f-6207e22d2a18" containerName="util"
Jul 14 17:39:13.585842 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585800    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be48f74-1675-42e1-a07f-6207e22d2a18" containerName="util"
Jul 14 17:39:13.585842 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585809    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7df75c1-34d5-4e45-8c2f-d30cc4d29d48" containerName="pull"
Jul 14 17:39:13.585842 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585814    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7df75c1-34d5-4e45-8c2f-d30cc4d29d48" containerName="pull"
Jul 14 17:39:13.585842 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585824    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b693041-1102-4307-ade2-6afd89206f18" containerName="util"
Jul 14 17:39:13.585842 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585829    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b693041-1102-4307-ade2-6afd89206f18" containerName="util"
Jul 14 17:39:13.585842 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585835    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b693041-1102-4307-ade2-6afd89206f18" containerName="pull"
Jul 14 17:39:13.585842 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585840    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b693041-1102-4307-ade2-6afd89206f18" containerName="pull"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585848    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7df75c1-34d5-4e45-8c2f-d30cc4d29d48" containerName="util"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585853    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7df75c1-34d5-4e45-8c2f-d30cc4d29d48" containerName="util"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585859    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24621efc-cea7-4df7-a5af-e02a4ab5a8d2" containerName="pull"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585864    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="24621efc-cea7-4df7-a5af-e02a4ab5a8d2" containerName="pull"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585873    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be48f74-1675-42e1-a07f-6207e22d2a18" containerName="extract"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585878    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be48f74-1675-42e1-a07f-6207e22d2a18" containerName="extract"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585887    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7df75c1-34d5-4e45-8c2f-d30cc4d29d48" containerName="extract"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585892    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7df75c1-34d5-4e45-8c2f-d30cc4d29d48" containerName="extract"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585898    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b693041-1102-4307-ade2-6afd89206f18" containerName="extract"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585902    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b693041-1102-4307-ade2-6afd89206f18" containerName="extract"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585908    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24621efc-cea7-4df7-a5af-e02a4ab5a8d2" containerName="util"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585918    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="24621efc-cea7-4df7-a5af-e02a4ab5a8d2" containerName="util"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585925    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24621efc-cea7-4df7-a5af-e02a4ab5a8d2" containerName="extract"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585930    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="24621efc-cea7-4df7-a5af-e02a4ab5a8d2" containerName="extract"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585936    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be48f74-1675-42e1-a07f-6207e22d2a18" containerName="pull"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585940    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be48f74-1675-42e1-a07f-6207e22d2a18" containerName="pull"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585989    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4be48f74-1675-42e1-a07f-6207e22d2a18" containerName="extract"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.585996    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="24621efc-cea7-4df7-a5af-e02a4ab5a8d2" containerName="extract"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.586005    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b693041-1102-4307-ade2-6afd89206f18" containerName="extract"
Jul 14 17:39:13.586117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.586012    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7df75c1-34d5-4e45-8c2f-d30cc4d29d48" containerName="extract"
Jul 14 17:39:13.593874 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.593853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"
Jul 14 17:39:13.597334 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.597310    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5ccf6\""
Jul 14 17:39:13.610774 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.610747    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"]
Jul 14 17:39:13.644292 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.644258    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsjsm\" (UniqueName: \"kubernetes.io/projected/187f66c7-c609-419a-9c98-df7b0ce2bf3e-kube-api-access-wsjsm\") pod \"limitador-operator-controller-manager-859d7b78bd-5sc65\" (UID: \"187f66c7-c609-419a-9c98-df7b0ce2bf3e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"
Jul 14 17:39:13.745087 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.745054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsjsm\" (UniqueName: \"kubernetes.io/projected/187f66c7-c609-419a-9c98-df7b0ce2bf3e-kube-api-access-wsjsm\") pod \"limitador-operator-controller-manager-859d7b78bd-5sc65\" (UID: \"187f66c7-c609-419a-9c98-df7b0ce2bf3e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"
Jul 14 17:39:13.759360 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.759328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsjsm\" (UniqueName: \"kubernetes.io/projected/187f66c7-c609-419a-9c98-df7b0ce2bf3e-kube-api-access-wsjsm\") pod \"limitador-operator-controller-manager-859d7b78bd-5sc65\" (UID: \"187f66c7-c609-419a-9c98-df7b0ce2bf3e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"
Jul 14 17:39:13.904032 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:13.903936    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"
Jul 14 17:39:14.058964 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:14.058922    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"]
Jul 14 17:39:14.062865 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:39:14.062830    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187f66c7_c609_419a_9c98_df7b0ce2bf3e.slice/crio-bab562e915b42adb9a927fc3c61158efac5d2aae2d59bb650b43cb1544600047 WatchSource:0}: Error finding container bab562e915b42adb9a927fc3c61158efac5d2aae2d59bb650b43cb1544600047: Status 404 returned error can't find the container with id bab562e915b42adb9a927fc3c61158efac5d2aae2d59bb650b43cb1544600047
Jul 14 17:39:14.627274 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:14.627238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65" event={"ID":"187f66c7-c609-419a-9c98-df7b0ce2bf3e","Type":"ContainerStarted","Data":"bab562e915b42adb9a927fc3c61158efac5d2aae2d59bb650b43cb1544600047"}
Jul 14 17:39:15.086476 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:15.086441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:16.242769 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:16.242680    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:16.640926 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:16.640830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65" event={"ID":"187f66c7-c609-419a-9c98-df7b0ce2bf3e","Type":"ContainerStarted","Data":"0b7d82e6bcaadfc1feef3d4e202026808885cc19c85cafc77aef6cdfff37966b"}
Jul 14 17:39:16.641103 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:16.641003    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"
Jul 14 17:39:16.667266 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:16.667193    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65" podStartSLOduration=1.739373657 podStartE2EDuration="3.667178842s" totalImagesPullingTime="1.927805185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:39:16.666053028 +0000 UTC m=+413.382446719" watchObservedRunningTime="2026-07-14 17:39:16.667178842 +0000 UTC m=+413.383572534"
Jul 14 17:39:17.337138 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:17.337098    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ffz54"]
Jul 14 17:39:17.339406 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:17.339387    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ffz54"
Jul 14 17:39:17.346594 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:17.346573    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 14 17:39:17.363487 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:17.363465    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jr7s9\""
Jul 14 17:39:17.369174 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:17.369140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmtgj\" (UniqueName: \"kubernetes.io/projected/f625fea1-c044-4e41-9652-2311ff48ac52-kube-api-access-qmtgj\") pod \"dns-operator-controller-manager-864dd78c4c-ffz54\" (UID: \"f625fea1-c044-4e41-9652-2311ff48ac52\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ffz54"
Jul 14 17:39:17.387229 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:17.387176    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ffz54"]
Jul 14 17:39:17.470116 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:17.470082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmtgj\" (UniqueName: \"kubernetes.io/projected/f625fea1-c044-4e41-9652-2311ff48ac52-kube-api-access-qmtgj\") pod \"dns-operator-controller-manager-864dd78c4c-ffz54\" (UID: \"f625fea1-c044-4e41-9652-2311ff48ac52\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ffz54"
Jul 14 17:39:17.499780 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:17.499735    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmtgj\" (UniqueName: \"kubernetes.io/projected/f625fea1-c044-4e41-9652-2311ff48ac52-kube-api-access-qmtgj\") pod \"dns-operator-controller-manager-864dd78c4c-ffz54\" (UID: \"f625fea1-c044-4e41-9652-2311ff48ac52\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ffz54"
Jul 14 17:39:17.650687 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:17.650608    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ffz54"
Jul 14 17:39:17.744470 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:17.744427    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:17.819360 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:17.819340    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ffz54"]
Jul 14 17:39:17.821663 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:39:17.821635    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf625fea1_c044_4e41_9652_2311ff48ac52.slice/crio-e899df70a015fdd3144f957edd6697e2c36740d085dea0d2ffb87e38a2a6a671 WatchSource:0}: Error finding container e899df70a015fdd3144f957edd6697e2c36740d085dea0d2ffb87e38a2a6a671: Status 404 returned error can't find the container with id e899df70a015fdd3144f957edd6697e2c36740d085dea0d2ffb87e38a2a6a671
Jul 14 17:39:18.269972 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:18.269940    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:18.485188 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:18.485160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:18.654884 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:18.654793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ffz54" event={"ID":"f625fea1-c044-4e41-9652-2311ff48ac52","Type":"ContainerStarted","Data":"e899df70a015fdd3144f957edd6697e2c36740d085dea0d2ffb87e38a2a6a671"}
Jul 14 17:39:19.966453 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:19.966420    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:21.040449 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:21.040399    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-jpqgl"]
Jul 14 17:39:21.043815 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:21.043791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jpqgl"
Jul 14 17:39:21.046874 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:21.046840    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6hf5d\""
Jul 14 17:39:21.065756 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:21.065687    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-jpqgl"]
Jul 14 17:39:21.097053 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:21.097010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59f7\" (UniqueName: \"kubernetes.io/projected/6fe9e15d-bb8e-4c0a-a39f-3f57397e01bd-kube-api-access-r59f7\") pod \"authorino-operator-75cc8d5cb7-jpqgl\" (UID: \"6fe9e15d-bb8e-4c0a-a39f-3f57397e01bd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-jpqgl"
Jul 14 17:39:21.198375 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:21.198317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59f7\" (UniqueName: \"kubernetes.io/projected/6fe9e15d-bb8e-4c0a-a39f-3f57397e01bd-kube-api-access-r59f7\") pod \"authorino-operator-75cc8d5cb7-jpqgl\" (UID: \"6fe9e15d-bb8e-4c0a-a39f-3f57397e01bd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-jpqgl"
Jul 14 17:39:21.220180 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:21.220152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59f7\" (UniqueName: \"kubernetes.io/projected/6fe9e15d-bb8e-4c0a-a39f-3f57397e01bd-kube-api-access-r59f7\") pod \"authorino-operator-75cc8d5cb7-jpqgl\" (UID: \"6fe9e15d-bb8e-4c0a-a39f-3f57397e01bd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-jpqgl"
Jul 14 17:39:21.357828 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:21.357748    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jpqgl"
Jul 14 17:39:21.517403 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:21.517379    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-jpqgl"]
Jul 14 17:39:21.519480 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:39:21.519452    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe9e15d_bb8e_4c0a_a39f_3f57397e01bd.slice/crio-d22017f4680c251c18ee5630df8f8fa149796849efb742e857898b180b654c19 WatchSource:0}: Error finding container d22017f4680c251c18ee5630df8f8fa149796849efb742e857898b180b654c19: Status 404 returned error can't find the container with id d22017f4680c251c18ee5630df8f8fa149796849efb742e857898b180b654c19
Jul 14 17:39:21.671807 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:21.671774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jpqgl" event={"ID":"6fe9e15d-bb8e-4c0a-a39f-3f57397e01bd","Type":"ContainerStarted","Data":"d22017f4680c251c18ee5630df8f8fa149796849efb742e857898b180b654c19"}
Jul 14 17:39:21.699029 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:21.698997    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:22.559895 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:22.559837    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:22.679851 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:22.679810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ffz54" event={"ID":"f625fea1-c044-4e41-9652-2311ff48ac52","Type":"ContainerStarted","Data":"b9e408bce8773144ce3c5698e34059bf2412d20e720df46b0ff9fbe82197e284"}
Jul 14 17:39:22.680045 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:22.680028    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ffz54"
Jul 14 17:39:22.712400 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:22.712348    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ffz54" podStartSLOduration=1.369645177 podStartE2EDuration="5.712332349s" totalImagesPullingTime="4.342687172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:39:22.709459379 +0000 UTC m=+419.425853070" watchObservedRunningTime="2026-07-14 17:39:22.712332349 +0000 UTC m=+419.428726086"
Jul 14 17:39:24.692279 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:24.692242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jpqgl" event={"ID":"6fe9e15d-bb8e-4c0a-a39f-3f57397e01bd","Type":"ContainerStarted","Data":"5c03995229493c039f18e12041bd09f070e5bca60222b2a1bd9b38108f7875dc"}
Jul 14 17:39:24.692659 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:24.692322    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jpqgl"
Jul 14 17:39:24.732331 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:24.732287    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jpqgl" podStartSLOduration=1.319802839 podStartE2EDuration="3.732271927s" totalImagesPullingTime="2.412469088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:39:24.7310863 +0000 UTC m=+421.447479992" watchObservedRunningTime="2026-07-14 17:39:24.732271927 +0000 UTC m=+421.448665618"
Jul 14 17:39:25.936651 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:25.936614    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:27.089754 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.089720    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"]
Jul 14 17:39:27.092135 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.092118    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:39:27.094702 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.094682    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vqlkv\""
Jul 14 17:39:27.111678 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.111652    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"]
Jul 14 17:39:27.151307 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.151272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mhzw\" (UniqueName: \"kubernetes.io/projected/47d468f9-10c5-4362-8663-98f06e935a3a-kube-api-access-5mhzw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gdkj7\" (UID: \"47d468f9-10c5-4362-8663-98f06e935a3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:39:27.151465 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.151317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47d468f9-10c5-4362-8663-98f06e935a3a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gdkj7\" (UID: \"47d468f9-10c5-4362-8663-98f06e935a3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:39:27.252763 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.252725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mhzw\" (UniqueName: \"kubernetes.io/projected/47d468f9-10c5-4362-8663-98f06e935a3a-kube-api-access-5mhzw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gdkj7\" (UID: \"47d468f9-10c5-4362-8663-98f06e935a3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:39:27.252956 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.252768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47d468f9-10c5-4362-8663-98f06e935a3a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gdkj7\" (UID: \"47d468f9-10c5-4362-8663-98f06e935a3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:39:27.253245 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.253184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47d468f9-10c5-4362-8663-98f06e935a3a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gdkj7\" (UID: \"47d468f9-10c5-4362-8663-98f06e935a3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:39:27.262094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.262072    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mhzw\" (UniqueName: \"kubernetes.io/projected/47d468f9-10c5-4362-8663-98f06e935a3a-kube-api-access-5mhzw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gdkj7\" (UID: \"47d468f9-10c5-4362-8663-98f06e935a3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:39:27.402373 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.402282    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:39:27.552070 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.552039    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"]
Jul 14 17:39:27.553129 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:39:27.553104    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d468f9_10c5_4362_8663_98f06e935a3a.slice/crio-be88e240d7637c2fd0b51e00cf19c3e9320d96e60402945757ccd7c6e1d38ec8 WatchSource:0}: Error finding container be88e240d7637c2fd0b51e00cf19c3e9320d96e60402945757ccd7c6e1d38ec8: Status 404 returned error can't find the container with id be88e240d7637c2fd0b51e00cf19c3e9320d96e60402945757ccd7c6e1d38ec8
Jul 14 17:39:27.650831 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.650802    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"
Jul 14 17:39:27.714499 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.714461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7" event={"ID":"47d468f9-10c5-4362-8663-98f06e935a3a","Type":"ContainerStarted","Data":"be88e240d7637c2fd0b51e00cf19c3e9320d96e60402945757ccd7c6e1d38ec8"}
Jul 14 17:39:27.736446 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.736400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:27.860679 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:27.860640    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:29.422136 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:29.422101    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:29.549909 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:29.549870    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:33.688661 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:33.688623    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ffz54"
Jul 14 17:39:35.705432 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:35.705410    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jpqgl"
Jul 14 17:39:35.730858 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:35.730824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:35.901052 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:35.901006    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:36.050971 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:36.050877    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:36.772682 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:36.772636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7" event={"ID":"47d468f9-10c5-4362-8663-98f06e935a3a","Type":"ContainerStarted","Data":"03acf6d7ef1da5a0b86c089351f8fabb3aec74e466675b569ef977e2c443792e"}
Jul 14 17:39:36.773215 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:36.772734    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:39:36.821641 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:36.821584    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7" podStartSLOduration=1.681890949 podStartE2EDuration="9.821568255s" totalImagesPullingTime="8.139677306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:39:36.817835098 +0000 UTC m=+433.534228789" watchObservedRunningTime="2026-07-14 17:39:36.821568255 +0000 UTC m=+433.537961947"
Jul 14 17:39:38.945249 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:38.945192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:38.971286 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:38.971255    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps"]
Jul 14 17:39:38.974183 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:38.974165    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps"
Jul 14 17:39:38.978473 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:38.978452    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 14 17:39:38.978565 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:38.978549    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 14 17:39:38.978768 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:38.978754    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ms5zj\""
Jul 14 17:39:38.995905 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:38.995875    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps"]
Jul 14 17:39:39.054200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:39.054169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1803e1bd-4131-42bb-844a-4ef466c6ddcd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rhrps\" (UID: \"1803e1bd-4131-42bb-844a-4ef466c6ddcd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps"
Jul 14 17:39:39.054200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:39.054220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1803e1bd-4131-42bb-844a-4ef466c6ddcd-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rhrps\" (UID: \"1803e1bd-4131-42bb-844a-4ef466c6ddcd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps"
Jul 14 17:39:39.054438 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:39.054324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlqkr\" (UniqueName: \"kubernetes.io/projected/1803e1bd-4131-42bb-844a-4ef466c6ddcd-kube-api-access-hlqkr\") pod \"kuadrant-console-plugin-66b5774985-rhrps\" (UID: \"1803e1bd-4131-42bb-844a-4ef466c6ddcd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps"
Jul 14 17:39:39.155136 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:39.155099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1803e1bd-4131-42bb-844a-4ef466c6ddcd-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rhrps\" (UID: \"1803e1bd-4131-42bb-844a-4ef466c6ddcd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps"
Jul 14 17:39:39.155325 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:39.155195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlqkr\" (UniqueName: \"kubernetes.io/projected/1803e1bd-4131-42bb-844a-4ef466c6ddcd-kube-api-access-hlqkr\") pod \"kuadrant-console-plugin-66b5774985-rhrps\" (UID: \"1803e1bd-4131-42bb-844a-4ef466c6ddcd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps"
Jul 14 17:39:39.155325 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:39.155287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1803e1bd-4131-42bb-844a-4ef466c6ddcd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rhrps\" (UID: \"1803e1bd-4131-42bb-844a-4ef466c6ddcd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps"
Jul 14 17:39:39.155409 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:39:39.155393    2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 14 17:39:39.155491 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:39:39.155481    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1803e1bd-4131-42bb-844a-4ef466c6ddcd-plugin-serving-cert podName:1803e1bd-4131-42bb-844a-4ef466c6ddcd nodeName:}" failed. No retries permitted until 2026-07-14 17:39:39.655457785 +0000 UTC m=+436.371851472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/1803e1bd-4131-42bb-844a-4ef466c6ddcd-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-rhrps" (UID: "1803e1bd-4131-42bb-844a-4ef466c6ddcd") : secret "plugin-serving-cert" not found
Jul 14 17:39:39.156281 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:39.156265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1803e1bd-4131-42bb-844a-4ef466c6ddcd-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rhrps\" (UID: \"1803e1bd-4131-42bb-844a-4ef466c6ddcd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps"
Jul 14 17:39:39.167036 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:39.167003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlqkr\" (UniqueName: \"kubernetes.io/projected/1803e1bd-4131-42bb-844a-4ef466c6ddcd-kube-api-access-hlqkr\") pod \"kuadrant-console-plugin-66b5774985-rhrps\" (UID: \"1803e1bd-4131-42bb-844a-4ef466c6ddcd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps"
Jul 14 17:39:39.295648 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:39.295568    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:39.659907 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:39.659832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1803e1bd-4131-42bb-844a-4ef466c6ddcd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rhrps\" (UID: \"1803e1bd-4131-42bb-844a-4ef466c6ddcd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps"
Jul 14 17:39:39.662410 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:39.662387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1803e1bd-4131-42bb-844a-4ef466c6ddcd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rhrps\" (UID: \"1803e1bd-4131-42bb-844a-4ef466c6ddcd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps"
Jul 14 17:39:39.713016 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:39.712984    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:39.811809 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:39.811774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:39.883799 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:39.883769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps"
Jul 14 17:39:40.040746 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:40.040722    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps"]
Jul 14 17:39:40.042280 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:39:40.042251    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1803e1bd_4131_42bb_844a_4ef466c6ddcd.slice/crio-9413c62594b7d18df4fc35f3fd667813fa53ce6811e8c9a39b483f1921ccad3f WatchSource:0}: Error finding container 9413c62594b7d18df4fc35f3fd667813fa53ce6811e8c9a39b483f1921ccad3f: Status 404 returned error can't find the container with id 9413c62594b7d18df4fc35f3fd667813fa53ce6811e8c9a39b483f1921ccad3f
Jul 14 17:39:40.793757 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:40.793718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps" event={"ID":"1803e1bd-4131-42bb-844a-4ef466c6ddcd","Type":"ContainerStarted","Data":"9413c62594b7d18df4fc35f3fd667813fa53ce6811e8c9a39b483f1921ccad3f"}
Jul 14 17:39:45.553262 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:45.553119    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:46.453145 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:46.453095    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:46.595829 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:46.595792    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:47.783100 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:47.783074    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:39:50.254700 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.254667    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"]
Jul 14 17:39:50.255222 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.254914    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7" podUID="47d468f9-10c5-4362-8663-98f06e935a3a" containerName="manager" containerID="cri-o://03acf6d7ef1da5a0b86c089351f8fabb3aec74e466675b569ef977e2c443792e" gracePeriod=2
Jul 14 17:39:50.279433 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.279403    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"]
Jul 14 17:39:50.317391 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.317362    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"]
Jul 14 17:39:50.317623 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.317569    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65" podUID="187f66c7-c609-419a-9c98-df7b0ce2bf3e" containerName="manager" containerID="cri-o://0b7d82e6bcaadfc1feef3d4e202026808885cc19c85cafc77aef6cdfff37966b" gracePeriod=2
Jul 14 17:39:50.337865 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.337837    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"]
Jul 14 17:39:50.354569 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:39:50.354519    2578 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"187f66c7-c609-419a-9c98-df7b0ce2bf3e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-5sc65\": pods \"limitador-operator-controller-manager-859d7b78bd-5sc65\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"
Jul 14 17:39:50.357105 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:39:50.357071    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gdkj7\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="47d468f9-10c5-4362-8663-98f06e935a3a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:39:50.395687 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.395645    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hfcjm"]
Jul 14 17:39:50.396005 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.395991    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="187f66c7-c609-419a-9c98-df7b0ce2bf3e" containerName="manager"
Jul 14 17:39:50.396005 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.396006    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="187f66c7-c609-419a-9c98-df7b0ce2bf3e" containerName="manager"
Jul 14 17:39:50.396103 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.396020    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d468f9-10c5-4362-8663-98f06e935a3a" containerName="manager"
Jul 14 17:39:50.396103 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.396025    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d468f9-10c5-4362-8663-98f06e935a3a" containerName="manager"
Jul 14 17:39:50.396103 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.396081    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="47d468f9-10c5-4362-8663-98f06e935a3a" containerName="manager"
Jul 14 17:39:50.396103 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.396091    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="187f66c7-c609-419a-9c98-df7b0ce2bf3e" containerName="manager"
Jul 14 17:39:50.399741 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.399718    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hfcjm"
Jul 14 17:39:50.416076 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:39:50.416041    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gdkj7\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="47d468f9-10c5-4362-8663-98f06e935a3a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:39:50.419425 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:39:50.419389    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5sc65\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="187f66c7-c609-419a-9c98-df7b0ce2bf3e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"
Jul 14 17:39:50.422024 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.421994    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hfcjm"]
Jul 14 17:39:50.460558 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.460528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp6f7\" (UniqueName: \"kubernetes.io/projected/c3c1b6d8-3a4d-4205-8df9-104feee899c4-kube-api-access-gp6f7\") pod \"limitador-operator-controller-manager-859d7b78bd-hfcjm\" (UID: \"c3c1b6d8-3a4d-4205-8df9-104feee899c4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hfcjm"
Jul 14 17:39:50.561970 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.561929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp6f7\" (UniqueName: \"kubernetes.io/projected/c3c1b6d8-3a4d-4205-8df9-104feee899c4-kube-api-access-gp6f7\") pod \"limitador-operator-controller-manager-859d7b78bd-hfcjm\" (UID: \"c3c1b6d8-3a4d-4205-8df9-104feee899c4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hfcjm"
Jul 14 17:39:50.576345 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.576308    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp6f7\" (UniqueName: \"kubernetes.io/projected/c3c1b6d8-3a4d-4205-8df9-104feee899c4-kube-api-access-gp6f7\") pod \"limitador-operator-controller-manager-859d7b78bd-hfcjm\" (UID: \"c3c1b6d8-3a4d-4205-8df9-104feee899c4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hfcjm"
Jul 14 17:39:50.713083 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.713047    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hfcjm"
Jul 14 17:39:50.853110 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.853031    2578 generic.go:356] "Generic (PLEG): container finished" podID="187f66c7-c609-419a-9c98-df7b0ce2bf3e" containerID="0b7d82e6bcaadfc1feef3d4e202026808885cc19c85cafc77aef6cdfff37966b" exitCode=0
Jul 14 17:39:50.855167 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:50.855129    2578 generic.go:356] "Generic (PLEG): container finished" podID="47d468f9-10c5-4362-8663-98f06e935a3a" containerID="03acf6d7ef1da5a0b86c089351f8fabb3aec74e466675b569ef977e2c443792e" exitCode=0
Jul 14 17:39:51.029802 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:51.029755    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:51.135797 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:51.135713    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:51.671927 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:51.671897    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"]
Jul 14 17:39:51.679200 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:51.679179    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"
Jul 14 17:39:51.683347 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:39:51.683314    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gdkj7\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="47d468f9-10c5-4362-8663-98f06e935a3a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:39:51.700028 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:51.699999    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"]
Jul 14 17:39:51.727818 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:39:51.727782    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5sc65\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="187f66c7-c609-419a-9c98-df7b0ce2bf3e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"
Jul 14 17:39:51.774339 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:51.774299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5858\" (UniqueName: \"kubernetes.io/projected/3c60a86b-9526-4023-951c-92dfd7819013-kube-api-access-v5858\") pod \"kuadrant-operator-controller-manager-86878c769f-b9xsj\" (UID: \"3c60a86b-9526-4023-951c-92dfd7819013\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"
Jul 14 17:39:51.774541 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:51.774390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c60a86b-9526-4023-951c-92dfd7819013-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b9xsj\" (UID: \"3c60a86b-9526-4023-951c-92dfd7819013\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"
Jul 14 17:39:51.875771 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:51.875739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5858\" (UniqueName: \"kubernetes.io/projected/3c60a86b-9526-4023-951c-92dfd7819013-kube-api-access-v5858\") pod \"kuadrant-operator-controller-manager-86878c769f-b9xsj\" (UID: \"3c60a86b-9526-4023-951c-92dfd7819013\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"
Jul 14 17:39:51.875954 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:51.875824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c60a86b-9526-4023-951c-92dfd7819013-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b9xsj\" (UID: \"3c60a86b-9526-4023-951c-92dfd7819013\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"
Jul 14 17:39:51.876273 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:51.876244    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c60a86b-9526-4023-951c-92dfd7819013-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b9xsj\" (UID: \"3c60a86b-9526-4023-951c-92dfd7819013\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"
Jul 14 17:39:51.895498 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:51.895467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5858\" (UniqueName: \"kubernetes.io/projected/3c60a86b-9526-4023-951c-92dfd7819013-kube-api-access-v5858\") pod \"kuadrant-operator-controller-manager-86878c769f-b9xsj\" (UID: \"3c60a86b-9526-4023-951c-92dfd7819013\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"
Jul 14 17:39:51.993136 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:51.993039    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"
Jul 14 17:39:55.268523 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:55.268429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:55.446898 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:55.446856    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:55.955655 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:55.955619    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:56.083476 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:56.083438    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:59.718033 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:59.717986    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:39:59.805990 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:39:59.805959    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:40:03.600518 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:03.600463    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:40:04.568397 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.568372    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:40:04.570949 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:40:04.570916    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gdkj7\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="47d468f9-10c5-4362-8663-98f06e935a3a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:40:04.697275 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.696990    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mhzw\" (UniqueName: \"kubernetes.io/projected/47d468f9-10c5-4362-8663-98f06e935a3a-kube-api-access-5mhzw\") pod \"47d468f9-10c5-4362-8663-98f06e935a3a\" (UID: \"47d468f9-10c5-4362-8663-98f06e935a3a\") "
Jul 14 17:40:04.697275 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.697147    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47d468f9-10c5-4362-8663-98f06e935a3a-extensions-socket-volume\") pod \"47d468f9-10c5-4362-8663-98f06e935a3a\" (UID: \"47d468f9-10c5-4362-8663-98f06e935a3a\") "
Jul 14 17:40:04.698522 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.698480    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d468f9-10c5-4362-8663-98f06e935a3a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "47d468f9-10c5-4362-8663-98f06e935a3a" (UID: "47d468f9-10c5-4362-8663-98f06e935a3a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:40:04.701010 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.700969    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d468f9-10c5-4362-8663-98f06e935a3a-kube-api-access-5mhzw" (OuterVolumeSpecName: "kube-api-access-5mhzw") pod "47d468f9-10c5-4362-8663-98f06e935a3a" (UID: "47d468f9-10c5-4362-8663-98f06e935a3a"). InnerVolumeSpecName "kube-api-access-5mhzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:40:04.704758 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.704669    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"
Jul 14 17:40:04.707680 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:40:04.707541    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gdkj7\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="47d468f9-10c5-4362-8663-98f06e935a3a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:40:04.709742 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:40:04.709681    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5sc65\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="187f66c7-c609-419a-9c98-df7b0ce2bf3e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"
Jul 14 17:40:04.790710 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:40:04.790673    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c1b6d8_3a4d_4205_8df9_104feee899c4.slice/crio-048423d09f861514d3710512c1934bdd99584c05d542a2061ddc99fa9a9e20ba WatchSource:0}: Error finding container 048423d09f861514d3710512c1934bdd99584c05d542a2061ddc99fa9a9e20ba: Status 404 returned error can't find the container with id 048423d09f861514d3710512c1934bdd99584c05d542a2061ddc99fa9a9e20ba
Jul 14 17:40:04.791570 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:40:04.791544    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c60a86b_9526_4023_951c_92dfd7819013.slice/crio-958890f458b8312035f53c0fa155b2d7fe11446c3b3b6f65aeff239cdc4348ea WatchSource:0}: Error finding container 958890f458b8312035f53c0fa155b2d7fe11446c3b3b6f65aeff239cdc4348ea: Status 404 returned error can't find the container with id 958890f458b8312035f53c0fa155b2d7fe11446c3b3b6f65aeff239cdc4348ea
Jul 14 17:40:04.800526 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.799862    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsjsm\" (UniqueName: \"kubernetes.io/projected/187f66c7-c609-419a-9c98-df7b0ce2bf3e-kube-api-access-wsjsm\") pod \"187f66c7-c609-419a-9c98-df7b0ce2bf3e\" (UID: \"187f66c7-c609-419a-9c98-df7b0ce2bf3e\") "
Jul 14 17:40:04.800526 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.800151    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47d468f9-10c5-4362-8663-98f06e935a3a-extensions-socket-volume\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:40:04.800526 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.800177    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mhzw\" (UniqueName: \"kubernetes.io/projected/47d468f9-10c5-4362-8663-98f06e935a3a-kube-api-access-5mhzw\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:40:04.801109 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.801083    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"]
Jul 14 17:40:04.803102 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.803054    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187f66c7-c609-419a-9c98-df7b0ce2bf3e-kube-api-access-wsjsm" (OuterVolumeSpecName: "kube-api-access-wsjsm") pod "187f66c7-c609-419a-9c98-df7b0ce2bf3e" (UID: "187f66c7-c609-419a-9c98-df7b0ce2bf3e"). InnerVolumeSpecName "kube-api-access-wsjsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:40:04.804192 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.804172    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hfcjm"]
Jul 14 17:40:04.900555 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.900521    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsjsm\" (UniqueName: \"kubernetes.io/projected/187f66c7-c609-419a-9c98-df7b0ce2bf3e-kube-api-access-wsjsm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:40:04.959121 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.959085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hfcjm" event={"ID":"c3c1b6d8-3a4d-4205-8df9-104feee899c4","Type":"ContainerStarted","Data":"9a6c264833e45bfb961a12b039fe180492d64cb41741ea1d7a524fe59ae57b8e"}
Jul 14 17:40:04.959294 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.959254    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hfcjm"
Jul 14 17:40:04.959372 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.959297    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hfcjm" event={"ID":"c3c1b6d8-3a4d-4205-8df9-104feee899c4","Type":"ContainerStarted","Data":"048423d09f861514d3710512c1934bdd99584c05d542a2061ddc99fa9a9e20ba"}
Jul 14 17:40:04.960819 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.960795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj" event={"ID":"3c60a86b-9526-4023-951c-92dfd7819013","Type":"ContainerStarted","Data":"0699b9e75a97309d354126136b1f66a7762f2897c3f0e6e28bd0046b2b3db2f4"}
Jul 14 17:40:04.960938 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.960834    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj" event={"ID":"3c60a86b-9526-4023-951c-92dfd7819013","Type":"ContainerStarted","Data":"958890f458b8312035f53c0fa155b2d7fe11446c3b3b6f65aeff239cdc4348ea"}
Jul 14 17:40:04.960938 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.960875    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"
Jul 14 17:40:04.961528 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:40:04.961507    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5sc65\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="187f66c7-c609-419a-9c98-df7b0ce2bf3e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"
Jul 14 17:40:04.962114 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.962099    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"
Jul 14 17:40:04.962175 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.962124    2578 scope.go:117] "RemoveContainer" containerID="0b7d82e6bcaadfc1feef3d4e202026808885cc19c85cafc77aef6cdfff37966b"
Jul 14 17:40:04.963774 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.963750    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps" event={"ID":"1803e1bd-4131-42bb-844a-4ef466c6ddcd","Type":"ContainerStarted","Data":"fbdede5bb9086e2a8738413217e186c8131f82e373ed177d8e675943467184c9"}
Jul 14 17:40:04.965264 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.965246    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:40:04.973467 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.973444    2578 scope.go:117] "RemoveContainer" containerID="03acf6d7ef1da5a0b86c089351f8fabb3aec74e466675b569ef977e2c443792e"
Jul 14 17:40:04.991658 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:04.991615    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hfcjm" podStartSLOduration=14.991596091 podStartE2EDuration="14.991596091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:40:04.990996211 +0000 UTC m=+461.707389903" watchObservedRunningTime="2026-07-14 17:40:04.991596091 +0000 UTC m=+461.707989783"
Jul 14 17:40:05.004429 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:40:05.004397    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gdkj7\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="47d468f9-10c5-4362-8663-98f06e935a3a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:40:05.007049 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:40:05.007024    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5sc65\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="187f66c7-c609-419a-9c98-df7b0ce2bf3e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5sc65"
Jul 14 17:40:05.041381 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:05.041327    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rhrps" podStartSLOduration=2.472566258 podStartE2EDuration="27.041311372s" totalImagesPullingTime="24.568745114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:40:05.040068771 +0000 UTC m=+461.756462465" watchObservedRunningTime="2026-07-14 17:40:05.041311372 +0000 UTC m=+461.757705063"
Jul 14 17:40:05.042117 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:40:05.042094    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gdkj7\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="47d468f9-10c5-4362-8663-98f06e935a3a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gdkj7"
Jul 14 17:40:05.084266 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:05.084141    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj" podStartSLOduration=14.084119377 podStartE2EDuration="14.084119377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:40:05.081082118 +0000 UTC m=+461.797475831" watchObservedRunningTime="2026-07-14 17:40:05.084119377 +0000 UTC m=+461.800513071"
Jul 14 17:40:05.782947 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:05.782912    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="187f66c7-c609-419a-9c98-df7b0ce2bf3e" path="/var/lib/kubelet/pods/187f66c7-c609-419a-9c98-df7b0ce2bf3e/volumes"
Jul 14 17:40:05.783315 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:05.783293    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47d468f9-10c5-4362-8663-98f06e935a3a" path="/var/lib/kubelet/pods/47d468f9-10c5-4362-8663-98f06e935a3a/volumes"
Jul 14 17:40:09.981839 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:09.981803    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:40:10.062565 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:10.062520    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:40:15.975733 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:15.975702    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hfcjm"
Jul 14 17:40:15.976144 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:15.975773    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"
Jul 14 17:40:33.030253 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:33.030196    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:40:33.157102 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:33.157070    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:40:38.424191 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:38.424160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:40:38.583912 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:38.583880    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:40:38.676902 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:38.676817    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:40:38.998757 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:38.998677    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:40:39.055021 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:39.054986    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:40:39.055279 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:39.055123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"
Jul 14 17:40:39.058719 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:39.058697    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 14 17:40:39.068182 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:39.068161    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:40:39.157684 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:39.157651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns4vm\" (UniqueName: \"kubernetes.io/projected/ca5f0f7f-bb5f-484b-8aa1-9754083cb27a-kube-api-access-ns4vm\") pod \"limitador-limitador-54d6d988d5-x7kzr\" (UID: \"ca5f0f7f-bb5f-484b-8aa1-9754083cb27a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"
Jul 14 17:40:39.157866 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:39.157693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca5f0f7f-bb5f-484b-8aa1-9754083cb27a-config-file\") pod \"limitador-limitador-54d6d988d5-x7kzr\" (UID: \"ca5f0f7f-bb5f-484b-8aa1-9754083cb27a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"
Jul 14 17:40:39.259226 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:39.259113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns4vm\" (UniqueName: \"kubernetes.io/projected/ca5f0f7f-bb5f-484b-8aa1-9754083cb27a-kube-api-access-ns4vm\") pod \"limitador-limitador-54d6d988d5-x7kzr\" (UID: \"ca5f0f7f-bb5f-484b-8aa1-9754083cb27a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"
Jul 14 17:40:39.259226 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:39.259154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca5f0f7f-bb5f-484b-8aa1-9754083cb27a-config-file\") pod \"limitador-limitador-54d6d988d5-x7kzr\" (UID: \"ca5f0f7f-bb5f-484b-8aa1-9754083cb27a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"
Jul 14 17:40:39.259753 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:39.259725    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca5f0f7f-bb5f-484b-8aa1-9754083cb27a-config-file\") pod \"limitador-limitador-54d6d988d5-x7kzr\" (UID: \"ca5f0f7f-bb5f-484b-8aa1-9754083cb27a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"
Jul 14 17:40:39.272823 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:39.272794    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns4vm\" (UniqueName: \"kubernetes.io/projected/ca5f0f7f-bb5f-484b-8aa1-9754083cb27a-kube-api-access-ns4vm\") pod \"limitador-limitador-54d6d988d5-x7kzr\" (UID: \"ca5f0f7f-bb5f-484b-8aa1-9754083cb27a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"
Jul 14 17:40:39.369117 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:39.369081    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"
Jul 14 17:40:39.529519 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:39.529495    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:40:39.532153 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:40:39.532126    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5f0f7f_bb5f_484b_8aa1_9754083cb27a.slice/crio-b45c409e723be2837cf7b9c27b636bcd06384bdbd0a14acc2d3f5fb5369f7c61 WatchSource:0}: Error finding container b45c409e723be2837cf7b9c27b636bcd06384bdbd0a14acc2d3f5fb5369f7c61: Status 404 returned error can't find the container with id b45c409e723be2837cf7b9c27b636bcd06384bdbd0a14acc2d3f5fb5369f7c61
Jul 14 17:40:39.563810 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:39.563750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:40:39.661656 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:39.661622    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:40:40.158219 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:40.158176    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-x7kzr" event={"ID":"ca5f0f7f-bb5f-484b-8aa1-9754083cb27a","Type":"ContainerStarted","Data":"b45c409e723be2837cf7b9c27b636bcd06384bdbd0a14acc2d3f5fb5369f7c61"}
Jul 14 17:40:40.241139 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:40.240819    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-5r68x"]
Jul 14 17:40:40.244927 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:40.244900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5r68x"
Jul 14 17:40:40.248681 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:40.248656    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9pp5t\""
Jul 14 17:40:40.258619 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:40.258593    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5r68x"]
Jul 14 17:40:40.267260 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:40.267232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sc24\" (UniqueName: \"kubernetes.io/projected/3f09d85e-0a55-41c2-8841-20eec245cdfc-kube-api-access-6sc24\") pod \"authorino-75d556bd77-5r68x\" (UID: \"3f09d85e-0a55-41c2-8841-20eec245cdfc\") " pod="kuadrant-system/authorino-75d556bd77-5r68x"
Jul 14 17:40:40.368830 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:40.368787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sc24\" (UniqueName: \"kubernetes.io/projected/3f09d85e-0a55-41c2-8841-20eec245cdfc-kube-api-access-6sc24\") pod \"authorino-75d556bd77-5r68x\" (UID: \"3f09d85e-0a55-41c2-8841-20eec245cdfc\") " pod="kuadrant-system/authorino-75d556bd77-5r68x"
Jul 14 17:40:40.392885 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:40.392851    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sc24\" (UniqueName: \"kubernetes.io/projected/3f09d85e-0a55-41c2-8841-20eec245cdfc-kube-api-access-6sc24\") pod \"authorino-75d556bd77-5r68x\" (UID: \"3f09d85e-0a55-41c2-8841-20eec245cdfc\") " pod="kuadrant-system/authorino-75d556bd77-5r68x"
Jul 14 17:40:40.564146 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:40.564106    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5r68x"
Jul 14 17:40:40.728910 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:40.728872    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5r68x"]
Jul 14 17:40:40.729609 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:40:40.729560    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f09d85e_0a55_41c2_8841_20eec245cdfc.slice/crio-8281b9a642111f373d5bc3538acc0886d42ccf996ab7cf66089375d4f13e500f WatchSource:0}: Error finding container 8281b9a642111f373d5bc3538acc0886d42ccf996ab7cf66089375d4f13e500f: Status 404 returned error can't find the container with id 8281b9a642111f373d5bc3538acc0886d42ccf996ab7cf66089375d4f13e500f
Jul 14 17:40:41.165742 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:41.165706    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5r68x" event={"ID":"3f09d85e-0a55-41c2-8841-20eec245cdfc","Type":"ContainerStarted","Data":"8281b9a642111f373d5bc3538acc0886d42ccf996ab7cf66089375d4f13e500f"}
Jul 14 17:40:43.183223 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:43.183164    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-x7kzr" event={"ID":"ca5f0f7f-bb5f-484b-8aa1-9754083cb27a","Type":"ContainerStarted","Data":"55ef2d17957a0639494f7220e4d6205fffee6374bb7b391e27abf037ac3f7579"}
Jul 14 17:40:43.183689 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:43.183315    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"
Jul 14 17:40:43.215767 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:43.215717    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-x7kzr" podStartSLOduration=2.315631087 podStartE2EDuration="5.21569816s" totalImagesPullingTime="2.900067073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:40:43.214263554 +0000 UTC m=+499.930657248" watchObservedRunningTime="2026-07-14 17:40:43.21569816 +0000 UTC m=+499.932091851"
Jul 14 17:40:45.064115 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:45.064082    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:40:45.181574 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:45.181535    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:40:45.197452 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:45.197422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5r68x" event={"ID":"3f09d85e-0a55-41c2-8841-20eec245cdfc","Type":"ContainerStarted","Data":"434f1e4cd4e1e31a226985eb6d15cb5bc446b0de1d277232893c96742b03e60e"}
Jul 14 17:40:45.226112 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:45.226051    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-5r68x" podStartSLOduration=1.499066934 podStartE2EDuration="5.226036218s" totalImagesPullingTime="3.726969284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:40:45.225248303 +0000 UTC m=+501.941642014" watchObservedRunningTime="2026-07-14 17:40:45.226036218 +0000 UTC m=+501.942429909"
Jul 14 17:40:54.190723 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:54.190690    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"
Jul 14 17:40:59.641124 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:59.641091    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:40:59.719173 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:40:59.719139    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:41:00.205060 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:00.205027    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:41:12.653044 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:12.653011    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:41:12.750416 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:12.750366    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:41:14.367835 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:14.367739    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5r68x"]
Jul 14 17:41:14.368312 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:14.367972    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-5r68x" podUID="3f09d85e-0a55-41c2-8841-20eec245cdfc" containerName="authorino" containerID="cri-o://434f1e4cd4e1e31a226985eb6d15cb5bc446b0de1d277232893c96742b03e60e" gracePeriod=30
Jul 14 17:41:14.637071 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:14.637045    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5r68x"
Jul 14 17:41:14.763343 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:14.763305    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sc24\" (UniqueName: \"kubernetes.io/projected/3f09d85e-0a55-41c2-8841-20eec245cdfc-kube-api-access-6sc24\") pod \"3f09d85e-0a55-41c2-8841-20eec245cdfc\" (UID: \"3f09d85e-0a55-41c2-8841-20eec245cdfc\") "
Jul 14 17:41:14.765534 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:14.765504    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f09d85e-0a55-41c2-8841-20eec245cdfc-kube-api-access-6sc24" (OuterVolumeSpecName: "kube-api-access-6sc24") pod "3f09d85e-0a55-41c2-8841-20eec245cdfc" (UID: "3f09d85e-0a55-41c2-8841-20eec245cdfc"). InnerVolumeSpecName "kube-api-access-6sc24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:41:14.864475 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:14.864431    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sc24\" (UniqueName: \"kubernetes.io/projected/3f09d85e-0a55-41c2-8841-20eec245cdfc-kube-api-access-6sc24\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:41:15.365101 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:15.365065    2578 generic.go:356] "Generic (PLEG): container finished" podID="3f09d85e-0a55-41c2-8841-20eec245cdfc" containerID="434f1e4cd4e1e31a226985eb6d15cb5bc446b0de1d277232893c96742b03e60e" exitCode=0
Jul 14 17:41:15.365320 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:15.365107    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5r68x"
Jul 14 17:41:15.365320 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:15.365143    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5r68x" event={"ID":"3f09d85e-0a55-41c2-8841-20eec245cdfc","Type":"ContainerDied","Data":"434f1e4cd4e1e31a226985eb6d15cb5bc446b0de1d277232893c96742b03e60e"}
Jul 14 17:41:15.365320 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:15.365183    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5r68x" event={"ID":"3f09d85e-0a55-41c2-8841-20eec245cdfc","Type":"ContainerDied","Data":"8281b9a642111f373d5bc3538acc0886d42ccf996ab7cf66089375d4f13e500f"}
Jul 14 17:41:15.365320 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:15.365199    2578 scope.go:117] "RemoveContainer" containerID="434f1e4cd4e1e31a226985eb6d15cb5bc446b0de1d277232893c96742b03e60e"
Jul 14 17:41:15.374471 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:15.374346    2578 scope.go:117] "RemoveContainer" containerID="434f1e4cd4e1e31a226985eb6d15cb5bc446b0de1d277232893c96742b03e60e"
Jul 14 17:41:15.374665 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:41:15.374605    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"434f1e4cd4e1e31a226985eb6d15cb5bc446b0de1d277232893c96742b03e60e\": container with ID starting with 434f1e4cd4e1e31a226985eb6d15cb5bc446b0de1d277232893c96742b03e60e not found: ID does not exist" containerID="434f1e4cd4e1e31a226985eb6d15cb5bc446b0de1d277232893c96742b03e60e"
Jul 14 17:41:15.374665 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:15.374629    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"434f1e4cd4e1e31a226985eb6d15cb5bc446b0de1d277232893c96742b03e60e"} err="failed to get container status \"434f1e4cd4e1e31a226985eb6d15cb5bc446b0de1d277232893c96742b03e60e\": rpc error: code = NotFound desc = could not find container \"434f1e4cd4e1e31a226985eb6d15cb5bc446b0de1d277232893c96742b03e60e\": container with ID starting with 434f1e4cd4e1e31a226985eb6d15cb5bc446b0de1d277232893c96742b03e60e not found: ID does not exist"
Jul 14 17:41:15.393923 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:15.393893    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5r68x"]
Jul 14 17:41:15.400446 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:15.400422    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5r68x"]
Jul 14 17:41:15.782890 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:15.782859    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f09d85e-0a55-41c2-8841-20eec245cdfc" path="/var/lib/kubelet/pods/3f09d85e-0a55-41c2-8841-20eec245cdfc/volumes"
Jul 14 17:41:19.278526 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:19.278485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:41:19.384249 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:19.384191    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:41:19.497844 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:19.497812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:41:22.725120 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:22.725062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:41:22.820154 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:22.820118    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:41:54.686337 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:54.686299    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:41:54.803727 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:54.803692    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:41:55.202027 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:55.201993    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:41:55.415457 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:55.415422    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:41:55.732673 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:55.732643    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:41:55.834163 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:55.834128    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:41:55.980127 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:41:55.980092    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:42:06.667507 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:06.667471    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:42:10.269405 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:10.269369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:42:13.583079 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:13.583044    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:42:14.055852 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:14.055817    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:42:17.159490 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:17.159449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:42:22.464901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:22.464865    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:42:22.568188 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:22.568157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:42:24.138578 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.138452    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"]
Jul 14 17:42:24.143526 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.139417    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f09d85e-0a55-41c2-8841-20eec245cdfc" containerName="authorino"
Jul 14 17:42:24.143526 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.139437    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f09d85e-0a55-41c2-8841-20eec245cdfc" containerName="authorino"
Jul 14 17:42:24.143526 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.139544    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f09d85e-0a55-41c2-8841-20eec245cdfc" containerName="authorino"
Jul 14 17:42:24.143526 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.142757    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.145495 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.145472    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 14 17:42:24.146442 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.146397    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 14 17:42:24.146669 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.146636    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 14 17:42:24.146945 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.146926    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6jbbn\""
Jul 14 17:42:24.155293 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.155266    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"]
Jul 14 17:42:24.155420 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.155344    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 17:42:24.155920 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.155899    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 17:42:24.175661 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.175629    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:42:24.244493 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.244458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.244493 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.244495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.244711 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.244538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.244711 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.244581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxt2n\" (UniqueName: \"kubernetes.io/projected/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-kube-api-access-mxt2n\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.244711 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.244637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.244711 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.244664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.345756 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.345719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.345756 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.345762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.345982 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.345809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.345982 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.345827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.345982 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.345864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.345982 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.345889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxt2n\" (UniqueName: \"kubernetes.io/projected/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-kube-api-access-mxt2n\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.346184 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.346158    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.346288 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.346250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.346365 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.346341    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.348185 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.348168    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.348368 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.348351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.354648 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.354621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxt2n\" (UniqueName: \"kubernetes.io/projected/a964ccaf-1a5b-49b2-b566-7c755e3dfc31-kube-api-access-mxt2n\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k78cm\" (UID: \"a964ccaf-1a5b-49b2-b566-7c755e3dfc31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.456520 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.456474    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:24.597720 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.597695    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"]
Jul 14 17:42:24.600379 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:42:24.600348    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda964ccaf_1a5b_49b2_b566_7c755e3dfc31.slice/crio-88d0bbbd1425bd88a1d0f031627fc48ead3c24029e73c7c681a0ca8121d0f56c WatchSource:0}: Error finding container 88d0bbbd1425bd88a1d0f031627fc48ead3c24029e73c7c681a0ca8121d0f56c: Status 404 returned error can't find the container with id 88d0bbbd1425bd88a1d0f031627fc48ead3c24029e73c7c681a0ca8121d0f56c
Jul 14 17:42:24.735378 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:24.735296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm" event={"ID":"a964ccaf-1a5b-49b2-b566-7c755e3dfc31","Type":"ContainerStarted","Data":"88d0bbbd1425bd88a1d0f031627fc48ead3c24029e73c7c681a0ca8121d0f56c"}
Jul 14 17:42:25.142883 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:25.142107    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:42:26.747383 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:26.747345    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm" event={"ID":"a964ccaf-1a5b-49b2-b566-7c755e3dfc31","Type":"ContainerStarted","Data":"d9ed72ebf9dfeb427affb0ab34090fb0a5efd3c68b1b57f97d72b645a17dba5d"}
Jul 14 17:42:26.769150 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:26.769070    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm" podStartSLOduration=1.6343550420000001 podStartE2EDuration="2.76905193s" totalImagesPullingTime="1.134696888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:42:26.767804561 +0000 UTC m=+603.484198253" watchObservedRunningTime="2026-07-14 17:42:26.76905193 +0000 UTC m=+603.485445624"
Jul 14 17:42:28.021263 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:28.021231    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:42:31.372172 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:31.372136    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:42:32.341325 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:32.341296    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:42:34.456709 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:34.456673    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:34.457094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:34.456756    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:34.471543 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:34.471518    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:34.801958 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:34.801859    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k78cm"
Jul 14 17:42:37.228628 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.228598    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"]
Jul 14 17:42:37.232328 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.232306    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.234872 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.234852    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 14 17:42:37.243416 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.243393    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"]
Jul 14 17:42:37.252630 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.252608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fed6a600-9415-4bdf-98a5-3725e69ffe93-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.252763 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.252639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fed6a600-9415-4bdf-98a5-3725e69ffe93-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.252763 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.252667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6lnk\" (UniqueName: \"kubernetes.io/projected/fed6a600-9415-4bdf-98a5-3725e69ffe93-kube-api-access-f6lnk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.252763 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.252754    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fed6a600-9415-4bdf-98a5-3725e69ffe93-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.252944 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.252820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fed6a600-9415-4bdf-98a5-3725e69ffe93-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.252944 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.252844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fed6a600-9415-4bdf-98a5-3725e69ffe93-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.267354 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.267328    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:42:37.353825 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.353788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fed6a600-9415-4bdf-98a5-3725e69ffe93-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.353825 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.353833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fed6a600-9415-4bdf-98a5-3725e69ffe93-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.354070 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.353855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fed6a600-9415-4bdf-98a5-3725e69ffe93-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.354070 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.353887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6lnk\" (UniqueName: \"kubernetes.io/projected/fed6a600-9415-4bdf-98a5-3725e69ffe93-kube-api-access-f6lnk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.354070 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.353923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fed6a600-9415-4bdf-98a5-3725e69ffe93-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.354070 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.353995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fed6a600-9415-4bdf-98a5-3725e69ffe93-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.354316 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.354258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fed6a600-9415-4bdf-98a5-3725e69ffe93-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.354377 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.354306    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fed6a600-9415-4bdf-98a5-3725e69ffe93-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.354377 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.354357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fed6a600-9415-4bdf-98a5-3725e69ffe93-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.356425 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.356407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fed6a600-9415-4bdf-98a5-3725e69ffe93-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.356608 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.356592    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fed6a600-9415-4bdf-98a5-3725e69ffe93-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.366485 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.366458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6lnk\" (UniqueName: \"kubernetes.io/projected/fed6a600-9415-4bdf-98a5-3725e69ffe93-kube-api-access-f6lnk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp\" (UID: \"fed6a600-9415-4bdf-98a5-3725e69ffe93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.543343 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.543265    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:37.685893 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.685864    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"]
Jul 14 17:42:37.687062 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:42:37.687034    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed6a600_9415_4bdf_98a5_3725e69ffe93.slice/crio-741f53700f8fe4c46809599d8e46689fe946f78f5e5f4b20add7890cf86e7cac WatchSource:0}: Error finding container 741f53700f8fe4c46809599d8e46689fe946f78f5e5f4b20add7890cf86e7cac: Status 404 returned error can't find the container with id 741f53700f8fe4c46809599d8e46689fe946f78f5e5f4b20add7890cf86e7cac
Jul 14 17:42:37.688446 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.688427    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 17:42:37.806596 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:37.806514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp" event={"ID":"fed6a600-9415-4bdf-98a5-3725e69ffe93","Type":"ContainerStarted","Data":"741f53700f8fe4c46809599d8e46689fe946f78f5e5f4b20add7890cf86e7cac"}
Jul 14 17:42:38.813425 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:38.813389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp" event={"ID":"fed6a600-9415-4bdf-98a5-3725e69ffe93","Type":"ContainerStarted","Data":"cc88b4c8e59d3864c77e73e0f29f4963348646e7584c5996254b569f355085aa"}
Jul 14 17:42:38.838052 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:38.838012    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:42:38.840086 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:38.840039    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp" podStartSLOduration=1.6419690770000002 podStartE2EDuration="1.840024805s" totalImagesPullingTime="198.055728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:42:38.835294113 +0000 UTC m=+615.551687822" watchObservedRunningTime="2026-07-14 17:42:38.840024805 +0000 UTC m=+615.556418493"
Jul 14 17:42:47.543588 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:47.543504    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:47.544042 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:47.543605    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:47.555854 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:47.555831    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:47.871142 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:47.871047    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp"
Jul 14 17:42:48.448652 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:48.448621    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:42:50.446310 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:50.446271    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:42:56.143484 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.143444    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"]
Jul 14 17:42:56.148320 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.148295    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.150947 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.150925    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 14 17:42:56.160897 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.160871    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"]
Jul 14 17:42:56.204183 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.204144    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:42:56.223703 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.223674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5b06c5-973f-40bd-8782-ce489f59e66d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.223820 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.223705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5b06c5-973f-40bd-8782-ce489f59e66d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.223820 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.223725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5b06c5-973f-40bd-8782-ce489f59e66d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.223890 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.223841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5b06c5-973f-40bd-8782-ce489f59e66d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.223890 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.223884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5b06c5-973f-40bd-8782-ce489f59e66d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.223958 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.223905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqc5h\" (UniqueName: \"kubernetes.io/projected/0a5b06c5-973f-40bd-8782-ce489f59e66d-kube-api-access-dqc5h\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.324767 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.324731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5b06c5-973f-40bd-8782-ce489f59e66d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.324942 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.324776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5b06c5-973f-40bd-8782-ce489f59e66d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.324942 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.324797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqc5h\" (UniqueName: \"kubernetes.io/projected/0a5b06c5-973f-40bd-8782-ce489f59e66d-kube-api-access-dqc5h\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.324942 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.324839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5b06c5-973f-40bd-8782-ce489f59e66d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.324942 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.324857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5b06c5-973f-40bd-8782-ce489f59e66d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.324942 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.324872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5b06c5-973f-40bd-8782-ce489f59e66d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.325192 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.325136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5b06c5-973f-40bd-8782-ce489f59e66d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.325286 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.325238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5b06c5-973f-40bd-8782-ce489f59e66d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.325349 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.325333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5b06c5-973f-40bd-8782-ce489f59e66d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.327192 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.327173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5b06c5-973f-40bd-8782-ce489f59e66d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.327503 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.327486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5b06c5-973f-40bd-8782-ce489f59e66d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.333629 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.333610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqc5h\" (UniqueName: \"kubernetes.io/projected/0a5b06c5-973f-40bd-8782-ce489f59e66d-kube-api-access-dqc5h\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m\" (UID: \"0a5b06c5-973f-40bd-8782-ce489f59e66d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.459912 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.459873    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:42:56.604377 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.604349    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"]
Jul 14 17:42:56.605308 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:42:56.605281    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a5b06c5_973f_40bd_8782_ce489f59e66d.slice/crio-2841ca4b0bf8e9045aa51c387cbdd712dfddf6256c47ca0e74e024abd28b888e WatchSource:0}: Error finding container 2841ca4b0bf8e9045aa51c387cbdd712dfddf6256c47ca0e74e024abd28b888e: Status 404 returned error can't find the container with id 2841ca4b0bf8e9045aa51c387cbdd712dfddf6256c47ca0e74e024abd28b888e
Jul 14 17:42:56.911409 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.911376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m" event={"ID":"0a5b06c5-973f-40bd-8782-ce489f59e66d","Type":"ContainerStarted","Data":"e9a3badadee80c6075eaa5e70b21a42f814f82054d949651562b82f3806d7f1c"}
Jul 14 17:42:56.911575 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.911417    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m" event={"ID":"0a5b06c5-973f-40bd-8782-ce489f59e66d","Type":"ContainerStarted","Data":"2841ca4b0bf8e9045aa51c387cbdd712dfddf6256c47ca0e74e024abd28b888e"}
Jul 14 17:42:56.934025 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:56.933980    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m" podStartSLOduration=0.751962051 podStartE2EDuration="933.96276ms" totalImagesPullingTime="182.000709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:42:56.933698983 +0000 UTC m=+633.650092680" watchObservedRunningTime="2026-07-14 17:42:56.93396276 +0000 UTC m=+633.650356452"
Jul 14 17:42:58.169890 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:42:58.169848    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:43:06.460819 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:06.460770    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:43:06.460819 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:06.460834    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:43:06.476894 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:06.476868    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:43:06.985540 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:06.985506    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m"
Jul 14 17:43:47.976352 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:47.976318    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85955854f9-7kr6n"]
Jul 14 17:43:47.979975 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:47.979951    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85955854f9-7kr6n"
Jul 14 17:43:47.983226 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:47.983183    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 14 17:43:47.983348 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:47.983183    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9pp5t\""
Jul 14 17:43:47.988944 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:47.988923    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85955854f9-7kr6n"]
Jul 14 17:43:48.096791 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:48.096752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j66fn\" (UniqueName: \"kubernetes.io/projected/9dd33441-61e5-4b5f-b2ff-875644fffc2c-kube-api-access-j66fn\") pod \"authorino-85955854f9-7kr6n\" (UID: \"9dd33441-61e5-4b5f-b2ff-875644fffc2c\") " pod="kuadrant-system/authorino-85955854f9-7kr6n"
Jul 14 17:43:48.096953 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:48.096803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9dd33441-61e5-4b5f-b2ff-875644fffc2c-tls-cert\") pod \"authorino-85955854f9-7kr6n\" (UID: \"9dd33441-61e5-4b5f-b2ff-875644fffc2c\") " pod="kuadrant-system/authorino-85955854f9-7kr6n"
Jul 14 17:43:48.197369 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:48.197334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j66fn\" (UniqueName: \"kubernetes.io/projected/9dd33441-61e5-4b5f-b2ff-875644fffc2c-kube-api-access-j66fn\") pod \"authorino-85955854f9-7kr6n\" (UID: \"9dd33441-61e5-4b5f-b2ff-875644fffc2c\") " pod="kuadrant-system/authorino-85955854f9-7kr6n"
Jul 14 17:43:48.197570 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:48.197392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9dd33441-61e5-4b5f-b2ff-875644fffc2c-tls-cert\") pod \"authorino-85955854f9-7kr6n\" (UID: \"9dd33441-61e5-4b5f-b2ff-875644fffc2c\") " pod="kuadrant-system/authorino-85955854f9-7kr6n"
Jul 14 17:43:48.202080 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:48.202046    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9dd33441-61e5-4b5f-b2ff-875644fffc2c-tls-cert\") pod \"authorino-85955854f9-7kr6n\" (UID: \"9dd33441-61e5-4b5f-b2ff-875644fffc2c\") " pod="kuadrant-system/authorino-85955854f9-7kr6n"
Jul 14 17:43:48.206378 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:48.206357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j66fn\" (UniqueName: \"kubernetes.io/projected/9dd33441-61e5-4b5f-b2ff-875644fffc2c-kube-api-access-j66fn\") pod \"authorino-85955854f9-7kr6n\" (UID: \"9dd33441-61e5-4b5f-b2ff-875644fffc2c\") " pod="kuadrant-system/authorino-85955854f9-7kr6n"
Jul 14 17:43:48.291003 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:48.290925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85955854f9-7kr6n"
Jul 14 17:43:48.336991 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:48.336960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:43:48.433765 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:48.433734    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85955854f9-7kr6n"]
Jul 14 17:43:48.435115 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:43:48.435072    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dd33441_61e5_4b5f_b2ff_875644fffc2c.slice/crio-f7be10edaa89d59fe2af4fa089a835b6bc3b9b9048ff6b8f7ffbec16a7ae8fd3 WatchSource:0}: Error finding container f7be10edaa89d59fe2af4fa089a835b6bc3b9b9048ff6b8f7ffbec16a7ae8fd3: Status 404 returned error can't find the container with id f7be10edaa89d59fe2af4fa089a835b6bc3b9b9048ff6b8f7ffbec16a7ae8fd3
Jul 14 17:43:48.435359 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:48.435137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:43:49.212045 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:49.212009    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85955854f9-7kr6n" event={"ID":"9dd33441-61e5-4b5f-b2ff-875644fffc2c","Type":"ContainerStarted","Data":"150b795219718beb1c8b4b243b58eb5df57717e3437144b43129fb9c63af4856"}
Jul 14 17:43:49.212483 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:49.212055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85955854f9-7kr6n" event={"ID":"9dd33441-61e5-4b5f-b2ff-875644fffc2c","Type":"ContainerStarted","Data":"f7be10edaa89d59fe2af4fa089a835b6bc3b9b9048ff6b8f7ffbec16a7ae8fd3"}
Jul 14 17:43:49.236787 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:49.236734    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85955854f9-7kr6n" podStartSLOduration=1.629145155 podStartE2EDuration="2.236718983s" totalImagesPullingTime="607.573828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:43:49.235966261 +0000 UTC m=+685.952359955" watchObservedRunningTime="2026-07-14 17:43:49.236718983 +0000 UTC m=+685.953112675"
Jul 14 17:43:49.637600 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:49.637559    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:43:49.741170 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:43:49.741135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:44:16.913942 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:44:16.913867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:44:18.046687 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:44:18.046651    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:44:28.032729 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:44:28.032693    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:44:40.595593 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:44:40.595557    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:44:48.558061 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:44:48.558029    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:44:59.531963 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:44:59.531922    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:45:01.841126 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:01.841095    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:45:05.207474 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:05.207441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:45:13.681569 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:13.681530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:45:26.651655 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:26.651623    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:45:33.725709 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:33.725672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:45:33.826357 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:33.826320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:45:37.039956 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:37.039921    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:45:43.939144 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:43.939053    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:45:44.048316 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:44.048276    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:45:44.139066 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:44.139029    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:45:50.736567 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:50.736533    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:45:50.839469 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:50.839439    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:45:50.950031 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:50.950000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:45:51.138445 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:51.138365    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:45:57.737737 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:57.737701    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:45:57.840143 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:45:57.840108    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:46:15.259278 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:46:15.259243    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:46:17.551524 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:46:17.551493    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:46:26.354886 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:46:26.354852    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:46:34.262378 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:46:34.262340    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:46:43.049378 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:46:43.049340    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:46:53.479156 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:46:53.479117    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:47:09.057645 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:47:09.057611    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:47:16.449633 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:47:16.449541    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:47:23.836225 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:47:23.836171    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:47:24.192893 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:47:24.192870    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 17:47:24.195057 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:47:24.193720    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 17:47:52.603906 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:47:52.603870    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:47:58.529798 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:47:58.529755    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:48:04.253747 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:48:04.253705    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:48:28.940068 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:48:28.940029    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:48:41.137568 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:48:41.137532    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:48:51.977170 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:48:51.977082    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:48:56.952367 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:48:56.952335    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:49:08.036582 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:49:08.036543    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:49:17.643954 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:49:17.643915    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:49:26.863292 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:49:26.863259    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:49:35.838493 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:49:35.838457    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:50:03.330446 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:03.330414    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"]
Jul 14 17:50:03.330934 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:03.330640    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj" podUID="3c60a86b-9526-4023-951c-92dfd7819013" containerName="manager" containerID="cri-o://0699b9e75a97309d354126136b1f66a7762f2897c3f0e6e28bd0046b2b3db2f4" gracePeriod=10
Jul 14 17:50:05.483926 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:05.483893    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"
Jul 14 17:50:05.573094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:05.572995    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5858\" (UniqueName: \"kubernetes.io/projected/3c60a86b-9526-4023-951c-92dfd7819013-kube-api-access-v5858\") pod \"3c60a86b-9526-4023-951c-92dfd7819013\" (UID: \"3c60a86b-9526-4023-951c-92dfd7819013\") "
Jul 14 17:50:05.573094 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:05.573066    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c60a86b-9526-4023-951c-92dfd7819013-extensions-socket-volume\") pod \"3c60a86b-9526-4023-951c-92dfd7819013\" (UID: \"3c60a86b-9526-4023-951c-92dfd7819013\") "
Jul 14 17:50:05.573461 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:05.573433    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c60a86b-9526-4023-951c-92dfd7819013-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3c60a86b-9526-4023-951c-92dfd7819013" (UID: "3c60a86b-9526-4023-951c-92dfd7819013"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:50:05.575351 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:05.575331    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c60a86b-9526-4023-951c-92dfd7819013-kube-api-access-v5858" (OuterVolumeSpecName: "kube-api-access-v5858") pod "3c60a86b-9526-4023-951c-92dfd7819013" (UID: "3c60a86b-9526-4023-951c-92dfd7819013"). InnerVolumeSpecName "kube-api-access-v5858". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:50:05.674664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:05.674613    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5858\" (UniqueName: \"kubernetes.io/projected/3c60a86b-9526-4023-951c-92dfd7819013-kube-api-access-v5858\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:50:05.674664 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:05.674657    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c60a86b-9526-4023-951c-92dfd7819013-extensions-socket-volume\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:50:06.344686 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:06.344653    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c60a86b-9526-4023-951c-92dfd7819013" containerID="0699b9e75a97309d354126136b1f66a7762f2897c3f0e6e28bd0046b2b3db2f4" exitCode=0
Jul 14 17:50:06.344861 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:06.344694    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"
Jul 14 17:50:06.344861 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:06.344697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj" event={"ID":"3c60a86b-9526-4023-951c-92dfd7819013","Type":"ContainerDied","Data":"0699b9e75a97309d354126136b1f66a7762f2897c3f0e6e28bd0046b2b3db2f4"}
Jul 14 17:50:06.344861 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:06.344748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj" event={"ID":"3c60a86b-9526-4023-951c-92dfd7819013","Type":"ContainerDied","Data":"958890f458b8312035f53c0fa155b2d7fe11446c3b3b6f65aeff239cdc4348ea"}
Jul 14 17:50:06.344861 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:06.344770    2578 scope.go:117] "RemoveContainer" containerID="0699b9e75a97309d354126136b1f66a7762f2897c3f0e6e28bd0046b2b3db2f4"
Jul 14 17:50:06.354550 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:06.354527    2578 scope.go:117] "RemoveContainer" containerID="0699b9e75a97309d354126136b1f66a7762f2897c3f0e6e28bd0046b2b3db2f4"
Jul 14 17:50:06.354799 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:50:06.354781    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0699b9e75a97309d354126136b1f66a7762f2897c3f0e6e28bd0046b2b3db2f4\": container with ID starting with 0699b9e75a97309d354126136b1f66a7762f2897c3f0e6e28bd0046b2b3db2f4 not found: ID does not exist" containerID="0699b9e75a97309d354126136b1f66a7762f2897c3f0e6e28bd0046b2b3db2f4"
Jul 14 17:50:06.354847 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:06.354808    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0699b9e75a97309d354126136b1f66a7762f2897c3f0e6e28bd0046b2b3db2f4"} err="failed to get container status \"0699b9e75a97309d354126136b1f66a7762f2897c3f0e6e28bd0046b2b3db2f4\": rpc error: code = NotFound desc = could not find container \"0699b9e75a97309d354126136b1f66a7762f2897c3f0e6e28bd0046b2b3db2f4\": container with ID starting with 0699b9e75a97309d354126136b1f66a7762f2897c3f0e6e28bd0046b2b3db2f4 not found: ID does not exist"
Jul 14 17:50:06.368408 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:06.368386    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"]
Jul 14 17:50:06.373948 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:06.373925    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b9xsj"]
Jul 14 17:50:07.783468 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:07.783429    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c60a86b-9526-4023-951c-92dfd7819013" path="/var/lib/kubelet/pods/3c60a86b-9526-4023-951c-92dfd7819013/volumes"
Jul 14 17:50:30.987779 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:50:30.987746    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:51:26.191284 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:51:26.191241    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:51:35.095441 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:51:35.095400    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:51:49.898156 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:51:49.898074    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:52:03.188445 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:52:03.188409    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:52:12.602156 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:52:12.602116    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:52:24.113824 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:52:24.113792    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:52:24.227944 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:52:24.227841    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 17:52:24.231898 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:52:24.229078    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 17:52:32.694276 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:52:32.694229    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:52:43.692243 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:52:43.692198    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:52:56.096838 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:52:56.096807    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:53:04.412535 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:53:04.412496    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:53:16.305564 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:53:16.305482    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:53:38.109261 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:53:38.109230    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:53:46.704467 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:53:46.704431    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:53:56.716885 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:53:56.716849    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:54:08.699493 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:54:08.699456    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:54:23.515167 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:54:23.515127    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:54:27.954176 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:54:27.954140    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:54:32.603532 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:54:32.603496    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:54:41.698101 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:54:41.698063    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:54:52.109424 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:54:52.109339    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:55:01.110979 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:55:01.110935    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:55:10.629662 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:55:10.629623    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:55:16.502602 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:55:16.502565    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:55:19.981022 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:55:19.980987    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:55:22.997928 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:55:22.997899    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:55:32.298468 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:55:32.298431    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:55:41.903262 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:55:41.903228    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:55:54.289974 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:55:54.289933    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:56:05.994621 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:56:05.994585    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:56:13.499886 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:56:13.499855    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:56:32.069423 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:56:32.069387    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:56:32.402312 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:56:32.402228    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:56:44.401384 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:56:44.401337    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:56:54.086874 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:56:54.086837    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:56:57.366602 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:56:57.366568    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:02.048155 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:02.048121    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:07.103300 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:07.103260    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:57:10.205756 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:10.205723    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:15.323476 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:15.323439    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:18.361799 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:18.361761    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:18.459361 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:18.459327    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:18.823974 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:18.823943    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:21.500129 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:21.500090    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:57:24.261857 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:24.261739    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 17:57:24.265875 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:24.264421    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 17:57:24.809745 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:24.809709    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:24.895325 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:24.895270    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:25.101744 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:25.101652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:25.176695 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:25.176665    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:26.219509 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.219478    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:26.317523 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.317493    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:26.379778 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.379743    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"]
Jul 14 17:57:26.380109 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.380097    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c60a86b-9526-4023-951c-92dfd7819013" containerName="manager"
Jul 14 17:57:26.380155 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.380111    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c60a86b-9526-4023-951c-92dfd7819013" containerName="manager"
Jul 14 17:57:26.380234 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.380223    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c60a86b-9526-4023-951c-92dfd7819013" containerName="manager"
Jul 14 17:57:26.382851 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.382834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"
Jul 14 17:57:26.385735 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.385715    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 17:57:26.385851 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.385717    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-09df0cc0-serving-cert\""
Jul 14 17:57:26.386600 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.386581    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xxs8g\""
Jul 14 17:57:26.386682 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.386624    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 17:57:26.394342 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.394316    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"]
Jul 14 17:57:26.505721 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.505645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46dzq\" (UniqueName: \"kubernetes.io/projected/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-kube-api-access-46dzq\") pod \"maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn\" (UID: \"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"
Jul 14 17:57:26.505721 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.505681    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-maas-api-tls\") pod \"maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn\" (UID: \"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"
Jul 14 17:57:26.505900 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.505747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-openshift-service-ca\") pod \"maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn\" (UID: \"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"
Jul 14 17:57:26.607195 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.607162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46dzq\" (UniqueName: \"kubernetes.io/projected/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-kube-api-access-46dzq\") pod \"maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn\" (UID: \"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"
Jul 14 17:57:26.607388 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.607197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-maas-api-tls\") pod \"maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn\" (UID: \"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"
Jul 14 17:57:26.607388 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.607270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-openshift-service-ca\") pod \"maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn\" (UID: \"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"
Jul 14 17:57:26.607846 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.607829    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-openshift-service-ca\") pod \"maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn\" (UID: \"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"
Jul 14 17:57:26.609758 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.609737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-maas-api-tls\") pod \"maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn\" (UID: \"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"
Jul 14 17:57:26.620406 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.620386    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46dzq\" (UniqueName: \"kubernetes.io/projected/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-kube-api-access-46dzq\") pod \"maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn\" (UID: \"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"
Jul 14 17:57:26.693397 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.693360    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"
Jul 14 17:57:26.839063 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.839033    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"]
Jul 14 17:57:26.840390 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:57:26.840356    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d5c5f0_758e_4090_a5ce_f5e26391b7fc.slice/crio-b380e216fc59f7e0dff76a89e1be0a5e35da3c971797963cae7a1e5b488325f7 WatchSource:0}: Error finding container b380e216fc59f7e0dff76a89e1be0a5e35da3c971797963cae7a1e5b488325f7: Status 404 returned error can't find the container with id b380e216fc59f7e0dff76a89e1be0a5e35da3c971797963cae7a1e5b488325f7
Jul 14 17:57:26.841716 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:26.841697    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 17:57:27.743281 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:27.743241    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn" event={"ID":"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc","Type":"ContainerStarted","Data":"b380e216fc59f7e0dff76a89e1be0a5e35da3c971797963cae7a1e5b488325f7"}
Jul 14 17:57:28.750927 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:28.750879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn" event={"ID":"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc","Type":"ContainerStarted","Data":"8cafd70e59e80deefc2704e9d3287250ab8b82a4f693982f5e2f69953c01d155"}
Jul 14 17:57:28.751501 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:28.750965    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"
Jul 14 17:57:28.769984 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:28.769936    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn" podStartSLOduration=1.018441295 podStartE2EDuration="2.769922421s" totalImagesPullingTime="1.751481126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:57:28.76903597 +0000 UTC m=+1505.485429668" watchObservedRunningTime="2026-07-14 17:57:28.769922421 +0000 UTC m=+1505.486316112"
Jul 14 17:57:29.080386 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:29.080341    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:29.185409 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:29.185376    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:31.257799 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:31.257761    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5"]
Jul 14 17:57:31.261413 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:31.261396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5"
Jul 14 17:57:31.264186 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:31.264165    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pd7w9\""
Jul 14 17:57:31.271618 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:31.271593    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5"]
Jul 14 17:57:31.352264 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:31.352191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19c51af-d6a1-4a60-a1b0-c0db9a84c501-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5\" (UID: \"f19c51af-d6a1-4a60-a1b0-c0db9a84c501\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5"
Jul 14 17:57:31.453454 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:31.453414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19c51af-d6a1-4a60-a1b0-c0db9a84c501-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5\" (UID: \"f19c51af-d6a1-4a60-a1b0-c0db9a84c501\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5"
Jul 14 17:57:31.454041 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:31.454023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19c51af-d6a1-4a60-a1b0-c0db9a84c501-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5\" (UID: \"f19c51af-d6a1-4a60-a1b0-c0db9a84c501\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5"
Jul 14 17:57:31.573522 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:31.573434    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5"
Jul 14 17:57:31.712701 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:31.712675    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5"]
Jul 14 17:57:31.714099 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:57:31.714073    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19c51af_d6a1_4a60_a1b0_c0db9a84c501.slice/crio-109d8016c5e5b71ddb69f5a6767bc3b26e896a23660883751de4038ad362e3ae WatchSource:0}: Error finding container 109d8016c5e5b71ddb69f5a6767bc3b26e896a23660883751de4038ad362e3ae: Status 404 returned error can't find the container with id 109d8016c5e5b71ddb69f5a6767bc3b26e896a23660883751de4038ad362e3ae
Jul 14 17:57:31.768799 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:31.768764    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5" event={"ID":"f19c51af-d6a1-4a60-a1b0-c0db9a84c501","Type":"ContainerStarted","Data":"109d8016c5e5b71ddb69f5a6767bc3b26e896a23660883751de4038ad362e3ae"}
Jul 14 17:57:33.290973 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:33.290923    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:57:34.763040 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:34.763010    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"
Jul 14 17:57:40.822644 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:40.822610    2578 generic.go:356] "Generic (PLEG): container finished" podID="f19c51af-d6a1-4a60-a1b0-c0db9a84c501" containerID="7448c05d46a4f0d7edec9b24de3c3ea71cc55fc308fa3a3d57a99c4aa63f7918" exitCode=0
Jul 14 17:57:40.823044 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:40.822665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5" event={"ID":"f19c51af-d6a1-4a60-a1b0-c0db9a84c501","Type":"ContainerDied","Data":"7448c05d46a4f0d7edec9b24de3c3ea71cc55fc308fa3a3d57a99c4aa63f7918"}
Jul 14 17:57:41.972940 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:41.972915    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5"
Jul 14 17:57:42.053500 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:42.053466    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19c51af-d6a1-4a60-a1b0-c0db9a84c501-maas-api-service-ca\") pod \"f19c51af-d6a1-4a60-a1b0-c0db9a84c501\" (UID: \"f19c51af-d6a1-4a60-a1b0-c0db9a84c501\") "
Jul 14 17:57:42.059758 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:42.054002    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19c51af-d6a1-4a60-a1b0-c0db9a84c501-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f19c51af-d6a1-4a60-a1b0-c0db9a84c501" (UID: "f19c51af-d6a1-4a60-a1b0-c0db9a84c501"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:57:42.059758 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:42.054165    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19c51af-d6a1-4a60-a1b0-c0db9a84c501-maas-api-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:57:42.835093 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:42.835063    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5"
Jul 14 17:57:42.835285 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:42.835064    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5" event={"ID":"f19c51af-d6a1-4a60-a1b0-c0db9a84c501","Type":"ContainerDied","Data":"109d8016c5e5b71ddb69f5a6767bc3b26e896a23660883751de4038ad362e3ae"}
Jul 14 17:57:42.835285 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:42.835191    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="109d8016c5e5b71ddb69f5a6767bc3b26e896a23660883751de4038ad362e3ae"
Jul 14 17:57:51.810240 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:51.810133    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"]
Jul 14 17:57:51.810776 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:51.810424    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn" podUID="e3d5c5f0-758e-4090-a5ce-f5e26391b7fc" containerName="maas-api" containerID="cri-o://8cafd70e59e80deefc2704e9d3287250ab8b82a4f693982f5e2f69953c01d155" gracePeriod=30
Jul 14 17:57:51.855964 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:51.855931    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:57:52.062779 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.062725    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"
Jul 14 17:57:52.244905 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.244873    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-maas-api-tls\") pod \"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc\" (UID: \"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc\") "
Jul 14 17:57:52.244905 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.244910    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-openshift-service-ca\") pod \"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc\" (UID: \"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc\") "
Jul 14 17:57:52.245132 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.244955    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46dzq\" (UniqueName: \"kubernetes.io/projected/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-kube-api-access-46dzq\") pod \"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc\" (UID: \"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc\") "
Jul 14 17:57:52.245321 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.245288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3d5c5f0-758e-4090-a5ce-f5e26391b7fc" (UID: "e3d5c5f0-758e-4090-a5ce-f5e26391b7fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:57:52.247304 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.247280    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3d5c5f0-758e-4090-a5ce-f5e26391b7fc" (UID: "e3d5c5f0-758e-4090-a5ce-f5e26391b7fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:57:52.247395 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.247374    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-kube-api-access-46dzq" (OuterVolumeSpecName: "kube-api-access-46dzq") pod "e3d5c5f0-758e-4090-a5ce-f5e26391b7fc" (UID: "e3d5c5f0-758e-4090-a5ce-f5e26391b7fc"). InnerVolumeSpecName "kube-api-access-46dzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:57:52.346389 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.346305    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-maas-api-tls\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:57:52.346389 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.346343    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-openshift-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:57:52.346389 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.346359    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46dzq\" (UniqueName: \"kubernetes.io/projected/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc-kube-api-access-46dzq\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:57:52.889445 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.889409    2578 generic.go:356] "Generic (PLEG): container finished" podID="e3d5c5f0-758e-4090-a5ce-f5e26391b7fc" containerID="8cafd70e59e80deefc2704e9d3287250ab8b82a4f693982f5e2f69953c01d155" exitCode=0
Jul 14 17:57:52.889875 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.889462    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"
Jul 14 17:57:52.889875 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.889500    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn" event={"ID":"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc","Type":"ContainerDied","Data":"8cafd70e59e80deefc2704e9d3287250ab8b82a4f693982f5e2f69953c01d155"}
Jul 14 17:57:52.889875 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.889562    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn" event={"ID":"e3d5c5f0-758e-4090-a5ce-f5e26391b7fc","Type":"ContainerDied","Data":"b380e216fc59f7e0dff76a89e1be0a5e35da3c971797963cae7a1e5b488325f7"}
Jul 14 17:57:52.889875 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.889593    2578 scope.go:117] "RemoveContainer" containerID="8cafd70e59e80deefc2704e9d3287250ab8b82a4f693982f5e2f69953c01d155"
Jul 14 17:57:52.899825 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.899810    2578 scope.go:117] "RemoveContainer" containerID="8cafd70e59e80deefc2704e9d3287250ab8b82a4f693982f5e2f69953c01d155"
Jul 14 17:57:52.900093 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:57:52.900074    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cafd70e59e80deefc2704e9d3287250ab8b82a4f693982f5e2f69953c01d155\": container with ID starting with 8cafd70e59e80deefc2704e9d3287250ab8b82a4f693982f5e2f69953c01d155 not found: ID does not exist" containerID="8cafd70e59e80deefc2704e9d3287250ab8b82a4f693982f5e2f69953c01d155"
Jul 14 17:57:52.900161 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.900102    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cafd70e59e80deefc2704e9d3287250ab8b82a4f693982f5e2f69953c01d155"} err="failed to get container status \"8cafd70e59e80deefc2704e9d3287250ab8b82a4f693982f5e2f69953c01d155\": rpc error: code = NotFound desc = could not find container \"8cafd70e59e80deefc2704e9d3287250ab8b82a4f693982f5e2f69953c01d155\": container with ID starting with 8cafd70e59e80deefc2704e9d3287250ab8b82a4f693982f5e2f69953c01d155 not found: ID does not exist"
Jul 14 17:57:52.912372 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.912335    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"]
Jul 14 17:57:52.915962 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:52.915939    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09df0cc0-685fcc585b-tvknn"]
Jul 14 17:57:53.784130 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:53.784096    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3d5c5f0-758e-4090-a5ce-f5e26391b7fc" path="/var/lib/kubelet/pods/e3d5c5f0-758e-4090-a5ce-f5e26391b7fc/volumes"
Jul 14 17:57:54.694460 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:54.694427    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:57:56.379381 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:57:56.379345    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:02.635086 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:02.635049    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:02.709176 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:02.709141    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:06.588121 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:06.588082    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:06.967790 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:06.967750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:07.081294 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:07.081261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:07.215615 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:07.215584    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:08.417996 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.417962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:08.496465 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.496432    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"]
Jul 14 17:58:08.496912 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.496891    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f19c51af-d6a1-4a60-a1b0-c0db9a84c501" containerName="revoke-keys"
Jul 14 17:58:08.497003 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.496915    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f19c51af-d6a1-4a60-a1b0-c0db9a84c501" containerName="revoke-keys"
Jul 14 17:58:08.497003 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.496937    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d5c5f0-758e-4090-a5ce-f5e26391b7fc" containerName="maas-api"
Jul 14 17:58:08.497003 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.496945    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d5c5f0-758e-4090-a5ce-f5e26391b7fc" containerName="maas-api"
Jul 14 17:58:08.497170 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.497068    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f19c51af-d6a1-4a60-a1b0-c0db9a84c501" containerName="revoke-keys"
Jul 14 17:58:08.497170 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.497086    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d5c5f0-758e-4090-a5ce-f5e26391b7fc" containerName="maas-api"
Jul 14 17:58:08.503330 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.503306    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"
Jul 14 17:58:08.506110 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.506090    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xxs8g\""
Jul 14 17:58:08.506411 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.506396    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b28b559b-serving-cert\""
Jul 14 17:58:08.506757 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.506744    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 17:58:08.506800 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.506784    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 17:58:08.511354 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.511037    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"]
Jul 14 17:58:08.547177 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.547150    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:08.593472 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.593437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a0a812-228d-441a-9105-56927907dead-openshift-service-ca\") pod \"maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs\" (UID: \"c4a0a812-228d-441a-9105-56927907dead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"
Jul 14 17:58:08.593472 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.593474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a0a812-228d-441a-9105-56927907dead-maas-api-tls\") pod \"maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs\" (UID: \"c4a0a812-228d-441a-9105-56927907dead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"
Jul 14 17:58:08.593700 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.593568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87c4s\" (UniqueName: \"kubernetes.io/projected/c4a0a812-228d-441a-9105-56927907dead-kube-api-access-87c4s\") pod \"maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs\" (UID: \"c4a0a812-228d-441a-9105-56927907dead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"
Jul 14 17:58:08.673616 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.673581    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:08.694797 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.694762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87c4s\" (UniqueName: \"kubernetes.io/projected/c4a0a812-228d-441a-9105-56927907dead-kube-api-access-87c4s\") pod \"maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs\" (UID: \"c4a0a812-228d-441a-9105-56927907dead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"
Jul 14 17:58:08.694971 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.694817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a0a812-228d-441a-9105-56927907dead-openshift-service-ca\") pod \"maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs\" (UID: \"c4a0a812-228d-441a-9105-56927907dead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"
Jul 14 17:58:08.694971 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.694846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a0a812-228d-441a-9105-56927907dead-maas-api-tls\") pod \"maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs\" (UID: \"c4a0a812-228d-441a-9105-56927907dead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"
Jul 14 17:58:08.695612 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.695595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a0a812-228d-441a-9105-56927907dead-openshift-service-ca\") pod \"maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs\" (UID: \"c4a0a812-228d-441a-9105-56927907dead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"
Jul 14 17:58:08.697551 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.697531    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a0a812-228d-441a-9105-56927907dead-maas-api-tls\") pod \"maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs\" (UID: \"c4a0a812-228d-441a-9105-56927907dead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"
Jul 14 17:58:08.704140 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.704114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87c4s\" (UniqueName: \"kubernetes.io/projected/c4a0a812-228d-441a-9105-56927907dead-kube-api-access-87c4s\") pod \"maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs\" (UID: \"c4a0a812-228d-441a-9105-56927907dead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"
Jul 14 17:58:08.816150 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.816114    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"
Jul 14 17:58:08.962754 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.962726    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"]
Jul 14 17:58:08.964707 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:58:08.964677    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a0a812_228d_441a_9105_56927907dead.slice/crio-ab540a569deb231ecf72a50c3cfb49e8ca31d9057116ee7d5b3eea69b9fcced1 WatchSource:0}: Error finding container ab540a569deb231ecf72a50c3cfb49e8ca31d9057116ee7d5b3eea69b9fcced1: Status 404 returned error can't find the container with id ab540a569deb231ecf72a50c3cfb49e8ca31d9057116ee7d5b3eea69b9fcced1
Jul 14 17:58:08.977789 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:08.977761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs" event={"ID":"c4a0a812-228d-441a-9105-56927907dead","Type":"ContainerStarted","Data":"ab540a569deb231ecf72a50c3cfb49e8ca31d9057116ee7d5b3eea69b9fcced1"}
Jul 14 17:58:09.985199 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:09.985163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs" event={"ID":"c4a0a812-228d-441a-9105-56927907dead","Type":"ContainerStarted","Data":"e77f4cdded71429bda517fee743a8c1d6824808761540c44d5f24b8e1bc35f4b"}
Jul 14 17:58:09.985610 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:09.985249    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"
Jul 14 17:58:10.003739 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:10.003694    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs" podStartSLOduration=1.421485348 podStartE2EDuration="2.003677097s" totalImagesPullingTime="582.191749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:58:10.001782621 +0000 UTC m=+1546.718176312" watchObservedRunningTime="2026-07-14 17:58:10.003677097 +0000 UTC m=+1546.720070788"
Jul 14 17:58:11.084646 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:11.084614    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:11.186051 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:11.186016    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:12.920160 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:12.920124    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx"]
Jul 14 17:58:12.924098 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:12.924075    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx"
Jul 14 17:58:12.927130 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:12.927107    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pd7w9\""
Jul 14 17:58:12.933524 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:12.933497    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx"]
Jul 14 17:58:13.029271 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:13.029241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd1d6f30-8634-4d95-89b8-8f9db5fcd7da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx\" (UID: \"dd1d6f30-8634-4d95-89b8-8f9db5fcd7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx"
Jul 14 17:58:13.130647 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:13.130563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd1d6f30-8634-4d95-89b8-8f9db5fcd7da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx\" (UID: \"dd1d6f30-8634-4d95-89b8-8f9db5fcd7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx"
Jul 14 17:58:13.131236 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:13.131199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd1d6f30-8634-4d95-89b8-8f9db5fcd7da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx\" (UID: \"dd1d6f30-8634-4d95-89b8-8f9db5fcd7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx"
Jul 14 17:58:13.236644 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:13.236613    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx"
Jul 14 17:58:13.376992 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:13.376959    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx"]
Jul 14 17:58:13.378499 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:58:13.378472    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1d6f30_8634_4d95_89b8_8f9db5fcd7da.slice/crio-5546793169f6dbfc5cd340c90a8a14c84bef45d0600f669812761cbeeae578cc WatchSource:0}: Error finding container 5546793169f6dbfc5cd340c90a8a14c84bef45d0600f669812761cbeeae578cc: Status 404 returned error can't find the container with id 5546793169f6dbfc5cd340c90a8a14c84bef45d0600f669812761cbeeae578cc
Jul 14 17:58:14.011565 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:14.011474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx" event={"ID":"dd1d6f30-8634-4d95-89b8-8f9db5fcd7da","Type":"ContainerStarted","Data":"b17ee1f7b12e123008e22eb644b6efba2827cb9d583a98695aa2d2e0804a7240"}
Jul 14 17:58:14.011565 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:14.011532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx" event={"ID":"dd1d6f30-8634-4d95-89b8-8f9db5fcd7da","Type":"ContainerStarted","Data":"5546793169f6dbfc5cd340c90a8a14c84bef45d0600f669812761cbeeae578cc"}
Jul 14 17:58:14.032591 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:14.032545    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx" podStartSLOduration=2.03252823 podStartE2EDuration="2.03252823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:58:14.030219681 +0000 UTC m=+1550.746613366" watchObservedRunningTime="2026-07-14 17:58:14.03252823 +0000 UTC m=+1550.748921921"
Jul 14 17:58:14.926968 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:14.926938    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:15.018732 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:15.018697    2578 generic.go:356] "Generic (PLEG): container finished" podID="dd1d6f30-8634-4d95-89b8-8f9db5fcd7da" containerID="b17ee1f7b12e123008e22eb644b6efba2827cb9d583a98695aa2d2e0804a7240" exitCode=7
Jul 14 17:58:15.019100 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:15.018782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx" event={"ID":"dd1d6f30-8634-4d95-89b8-8f9db5fcd7da","Type":"ContainerDied","Data":"b17ee1f7b12e123008e22eb644b6efba2827cb9d583a98695aa2d2e0804a7240"}
Jul 14 17:58:15.019100 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:15.018934    2578 scope.go:117] "RemoveContainer" containerID="b17ee1f7b12e123008e22eb644b6efba2827cb9d583a98695aa2d2e0804a7240"
Jul 14 17:58:15.030121 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:15.030095    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:15.998624 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:15.998596    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"
Jul 14 17:58:16.027355 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:16.027324    2578 generic.go:356] "Generic (PLEG): container finished" podID="dd1d6f30-8634-4d95-89b8-8f9db5fcd7da" containerID="5f8d70009d978ebac80654e7a44518f6c0cd652ea581840cef652164df9a5d3f" exitCode=7
Jul 14 17:58:16.027783 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:16.027415    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx" event={"ID":"dd1d6f30-8634-4d95-89b8-8f9db5fcd7da","Type":"ContainerDied","Data":"5f8d70009d978ebac80654e7a44518f6c0cd652ea581840cef652164df9a5d3f"}
Jul 14 17:58:16.027783 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:16.027480    2578 scope.go:117] "RemoveContainer" containerID="b17ee1f7b12e123008e22eb644b6efba2827cb9d583a98695aa2d2e0804a7240"
Jul 14 17:58:16.027783 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:16.027700    2578 scope.go:117] "RemoveContainer" containerID="5f8d70009d978ebac80654e7a44518f6c0cd652ea581840cef652164df9a5d3f"
Jul 14 17:58:16.027974 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:58:16.027951    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx_odh-ai-gateway-infra(dd1d6f30-8634-4d95-89b8-8f9db5fcd7da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx" podUID="dd1d6f30-8634-4d95-89b8-8f9db5fcd7da"
Jul 14 17:58:16.177971 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:16.177936    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:16.289822 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:16.289741    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:58:17.035873 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:17.035845    2578 scope.go:117] "RemoveContainer" containerID="5f8d70009d978ebac80654e7a44518f6c0cd652ea581840cef652164df9a5d3f"
Jul 14 17:58:17.036255 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:58:17.036063    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx_odh-ai-gateway-infra(dd1d6f30-8634-4d95-89b8-8f9db5fcd7da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx" podUID="dd1d6f30-8634-4d95-89b8-8f9db5fcd7da"
Jul 14 17:58:19.166627 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:19.166594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:19.228937 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:19.228904    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:58:32.777638 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:32.777607    2578 scope.go:117] "RemoveContainer" containerID="5f8d70009d978ebac80654e7a44518f6c0cd652ea581840cef652164df9a5d3f"
Jul 14 17:58:33.132011 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:33.131929    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx" event={"ID":"dd1d6f30-8634-4d95-89b8-8f9db5fcd7da","Type":"ContainerStarted","Data":"74f7f8fd48ef89a370924727d12ae257cff74400e909ee4ee998b592e62c19b7"}
Jul 14 17:58:34.139332 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:34.139294    2578 generic.go:356] "Generic (PLEG): container finished" podID="dd1d6f30-8634-4d95-89b8-8f9db5fcd7da" containerID="74f7f8fd48ef89a370924727d12ae257cff74400e909ee4ee998b592e62c19b7" exitCode=0
Jul 14 17:58:34.139870 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:34.139363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx" event={"ID":"dd1d6f30-8634-4d95-89b8-8f9db5fcd7da","Type":"ContainerDied","Data":"74f7f8fd48ef89a370924727d12ae257cff74400e909ee4ee998b592e62c19b7"}
Jul 14 17:58:34.139870 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:34.139435    2578 scope.go:117] "RemoveContainer" containerID="5f8d70009d978ebac80654e7a44518f6c0cd652ea581840cef652164df9a5d3f"
Jul 14 17:58:34.165507 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:34.165474    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx"]
Jul 14 17:58:35.298245 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:35.298194    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx"
Jul 14 17:58:35.432429 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:35.432392    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd1d6f30-8634-4d95-89b8-8f9db5fcd7da-maas-api-service-ca\") pod \"dd1d6f30-8634-4d95-89b8-8f9db5fcd7da\" (UID: \"dd1d6f30-8634-4d95-89b8-8f9db5fcd7da\") "
Jul 14 17:58:35.432752 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:35.432726    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1d6f30-8634-4d95-89b8-8f9db5fcd7da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd1d6f30-8634-4d95-89b8-8f9db5fcd7da" (UID: "dd1d6f30-8634-4d95-89b8-8f9db5fcd7da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:58:35.533642 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:35.533562    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd1d6f30-8634-4d95-89b8-8f9db5fcd7da-maas-api-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:58:36.154114 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:36.154040    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx"
Jul 14 17:58:36.154114 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:36.154071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx" event={"ID":"dd1d6f30-8634-4d95-89b8-8f9db5fcd7da","Type":"ContainerDied","Data":"5546793169f6dbfc5cd340c90a8a14c84bef45d0600f669812761cbeeae578cc"}
Jul 14 17:58:36.154353 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:36.154116    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5546793169f6dbfc5cd340c90a8a14c84bef45d0600f669812761cbeeae578cc"
Jul 14 17:58:36.183341 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:36.183311    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx"]
Jul 14 17:58:36.185472 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:36.185452    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-j2khx"]
Jul 14 17:58:37.783517 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:58:37.783487    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd1d6f30-8634-4d95-89b8-8f9db5fcd7da" path="/var/lib/kubelet/pods/dd1d6f30-8634-4d95-89b8-8f9db5fcd7da/volumes"
Jul 14 17:59:12.971368 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:12.971332    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb"]
Jul 14 17:59:12.971840 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:12.971726    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd1d6f30-8634-4d95-89b8-8f9db5fcd7da" containerName="revoke-keys"
Jul 14 17:59:12.971840 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:12.971740    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd1d6f30-8634-4d95-89b8-8f9db5fcd7da" containerName="revoke-keys"
Jul 14 17:59:12.971840 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:12.971764    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd1d6f30-8634-4d95-89b8-8f9db5fcd7da" containerName="revoke-keys"
Jul 14 17:59:12.971840 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:12.971770    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd1d6f30-8634-4d95-89b8-8f9db5fcd7da" containerName="revoke-keys"
Jul 14 17:59:12.971976 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:12.971848    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd1d6f30-8634-4d95-89b8-8f9db5fcd7da" containerName="revoke-keys"
Jul 14 17:59:12.971976 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:12.971858    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd1d6f30-8634-4d95-89b8-8f9db5fcd7da" containerName="revoke-keys"
Jul 14 17:59:12.974996 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:12.974979    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb"
Jul 14 17:59:12.977915 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:12.977896    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pd7w9\""
Jul 14 17:59:12.984819 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:12.984795    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb"]
Jul 14 17:59:13.064036 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:13.063995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99971710-4cf0-4461-810c-bf875fbfcafb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb\" (UID: \"99971710-4cf0-4461-810c-bf875fbfcafb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb"
Jul 14 17:59:13.071823 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:13.071792    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:13.165131 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:13.165093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99971710-4cf0-4461-810c-bf875fbfcafb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb\" (UID: \"99971710-4cf0-4461-810c-bf875fbfcafb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb"
Jul 14 17:59:13.165759 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:13.165737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99971710-4cf0-4461-810c-bf875fbfcafb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb\" (UID: \"99971710-4cf0-4461-810c-bf875fbfcafb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb"
Jul 14 17:59:13.171482 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:13.171456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:13.287425 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:13.287336    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb"
Jul 14 17:59:13.431751 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:13.431722    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb"]
Jul 14 17:59:13.432937 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:59:13.432912    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99971710_4cf0_4461_810c_bf875fbfcafb.slice/crio-22b5b926b7225ea33ddcfb1715ed3e4ef2daa4343908959822ce55413c18c956 WatchSource:0}: Error finding container 22b5b926b7225ea33ddcfb1715ed3e4ef2daa4343908959822ce55413c18c956: Status 404 returned error can't find the container with id 22b5b926b7225ea33ddcfb1715ed3e4ef2daa4343908959822ce55413c18c956
Jul 14 17:59:14.362745 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:14.362706    2578 generic.go:356] "Generic (PLEG): container finished" podID="99971710-4cf0-4461-810c-bf875fbfcafb" containerID="3e41acd4c615799cd45d7366cabf221c21db5efe7288adf713fd48a544cab957" exitCode=0
Jul 14 17:59:14.363162 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:14.362794    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb" event={"ID":"99971710-4cf0-4461-810c-bf875fbfcafb","Type":"ContainerDied","Data":"3e41acd4c615799cd45d7366cabf221c21db5efe7288adf713fd48a544cab957"}
Jul 14 17:59:14.363162 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:14.362866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb" event={"ID":"99971710-4cf0-4461-810c-bf875fbfcafb","Type":"ContainerStarted","Data":"22b5b926b7225ea33ddcfb1715ed3e4ef2daa4343908959822ce55413c18c956"}
Jul 14 17:59:15.497907 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:15.497878    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb"
Jul 14 17:59:15.590288 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:15.590253    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99971710-4cf0-4461-810c-bf875fbfcafb-maas-api-service-ca\") pod \"99971710-4cf0-4461-810c-bf875fbfcafb\" (UID: \"99971710-4cf0-4461-810c-bf875fbfcafb\") "
Jul 14 17:59:15.590612 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:15.590588    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99971710-4cf0-4461-810c-bf875fbfcafb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99971710-4cf0-4461-810c-bf875fbfcafb" (UID: "99971710-4cf0-4461-810c-bf875fbfcafb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:59:15.610033 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:15.610006    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:59:15.691284 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:15.691258    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99971710-4cf0-4461-810c-bf875fbfcafb-maas-api-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:59:16.374649 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:16.374611    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb"
Jul 14 17:59:16.374817 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:16.374669    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb" event={"ID":"99971710-4cf0-4461-810c-bf875fbfcafb","Type":"ContainerDied","Data":"22b5b926b7225ea33ddcfb1715ed3e4ef2daa4343908959822ce55413c18c956"}
Jul 14 17:59:16.374817 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:16.374718    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22b5b926b7225ea33ddcfb1715ed3e4ef2daa4343908959822ce55413c18c956"
Jul 14 17:59:19.450983 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:19.450950    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:19.917087 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:19.917006    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"]
Jul 14 17:59:19.917277 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:19.917254    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs" podUID="c4a0a812-228d-441a-9105-56927907dead" containerName="maas-api" containerID="cri-o://e77f4cdded71429bda517fee743a8c1d6824808761540c44d5f24b8e1bc35f4b" gracePeriod=30
Jul 14 17:59:19.947646 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:19.947610    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:20.095918 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.095883    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:20.174573 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.174552    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"
Jul 14 17:59:20.334985 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.334953    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87c4s\" (UniqueName: \"kubernetes.io/projected/c4a0a812-228d-441a-9105-56927907dead-kube-api-access-87c4s\") pod \"c4a0a812-228d-441a-9105-56927907dead\" (UID: \"c4a0a812-228d-441a-9105-56927907dead\") "
Jul 14 17:59:20.335198 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.335005    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a0a812-228d-441a-9105-56927907dead-maas-api-tls\") pod \"c4a0a812-228d-441a-9105-56927907dead\" (UID: \"c4a0a812-228d-441a-9105-56927907dead\") "
Jul 14 17:59:20.335198 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.335047    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a0a812-228d-441a-9105-56927907dead-openshift-service-ca\") pod \"c4a0a812-228d-441a-9105-56927907dead\" (UID: \"c4a0a812-228d-441a-9105-56927907dead\") "
Jul 14 17:59:20.335494 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.335458    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a0a812-228d-441a-9105-56927907dead-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4a0a812-228d-441a-9105-56927907dead" (UID: "c4a0a812-228d-441a-9105-56927907dead"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:59:20.337417 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.337393    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4a0a812-228d-441a-9105-56927907dead-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4a0a812-228d-441a-9105-56927907dead" (UID: "c4a0a812-228d-441a-9105-56927907dead"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:59:20.337504 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.337397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a0a812-228d-441a-9105-56927907dead-kube-api-access-87c4s" (OuterVolumeSpecName: "kube-api-access-87c4s") pod "c4a0a812-228d-441a-9105-56927907dead" (UID: "c4a0a812-228d-441a-9105-56927907dead"). InnerVolumeSpecName "kube-api-access-87c4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:59:20.399175 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.399146    2578 generic.go:356] "Generic (PLEG): container finished" podID="c4a0a812-228d-441a-9105-56927907dead" containerID="e77f4cdded71429bda517fee743a8c1d6824808761540c44d5f24b8e1bc35f4b" exitCode=0
Jul 14 17:59:20.399374 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.399195    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"
Jul 14 17:59:20.399374 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.399246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs" event={"ID":"c4a0a812-228d-441a-9105-56927907dead","Type":"ContainerDied","Data":"e77f4cdded71429bda517fee743a8c1d6824808761540c44d5f24b8e1bc35f4b"}
Jul 14 17:59:20.399374 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.399305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs" event={"ID":"c4a0a812-228d-441a-9105-56927907dead","Type":"ContainerDied","Data":"ab540a569deb231ecf72a50c3cfb49e8ca31d9057116ee7d5b3eea69b9fcced1"}
Jul 14 17:59:20.399374 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.399334    2578 scope.go:117] "RemoveContainer" containerID="e77f4cdded71429bda517fee743a8c1d6824808761540c44d5f24b8e1bc35f4b"
Jul 14 17:59:20.410907 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.410875    2578 scope.go:117] "RemoveContainer" containerID="e77f4cdded71429bda517fee743a8c1d6824808761540c44d5f24b8e1bc35f4b"
Jul 14 17:59:20.411442 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:59:20.411416    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e77f4cdded71429bda517fee743a8c1d6824808761540c44d5f24b8e1bc35f4b\": container with ID starting with e77f4cdded71429bda517fee743a8c1d6824808761540c44d5f24b8e1bc35f4b not found: ID does not exist" containerID="e77f4cdded71429bda517fee743a8c1d6824808761540c44d5f24b8e1bc35f4b"
Jul 14 17:59:20.411544 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.411452    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e77f4cdded71429bda517fee743a8c1d6824808761540c44d5f24b8e1bc35f4b"} err="failed to get container status \"e77f4cdded71429bda517fee743a8c1d6824808761540c44d5f24b8e1bc35f4b\": rpc error: code = NotFound desc = could not find container \"e77f4cdded71429bda517fee743a8c1d6824808761540c44d5f24b8e1bc35f4b\": container with ID starting with e77f4cdded71429bda517fee743a8c1d6824808761540c44d5f24b8e1bc35f4b not found: ID does not exist"
Jul 14 17:59:20.436397 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.436342    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87c4s\" (UniqueName: \"kubernetes.io/projected/c4a0a812-228d-441a-9105-56927907dead-kube-api-access-87c4s\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:59:20.436397 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.436364    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a0a812-228d-441a-9105-56927907dead-maas-api-tls\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:59:20.436397 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.436373    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a0a812-228d-441a-9105-56927907dead-openshift-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:59:20.456908 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.456881    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"]
Jul 14 17:59:20.466011 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:20.465989    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b28b559b-865d65cd6f-ct9fs"]
Jul 14 17:59:21.783750 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:21.783717    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4a0a812-228d-441a-9105-56927907dead" path="/var/lib/kubelet/pods/c4a0a812-228d-441a-9105-56927907dead/volumes"
Jul 14 17:59:26.436242 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:26.436185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:26.531306 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:26.531271    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:26.717236 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:26.717139    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:26.870996 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:26.870966    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:27.899081 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.899043    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:27.977457 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.977424    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"]
Jul 14 17:59:27.977796 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.977784    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4a0a812-228d-441a-9105-56927907dead" containerName="maas-api"
Jul 14 17:59:27.977850 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.977797    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4a0a812-228d-441a-9105-56927907dead" containerName="maas-api"
Jul 14 17:59:27.977850 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.977809    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd1d6f30-8634-4d95-89b8-8f9db5fcd7da" containerName="revoke-keys"
Jul 14 17:59:27.977850 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.977814    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd1d6f30-8634-4d95-89b8-8f9db5fcd7da" containerName="revoke-keys"
Jul 14 17:59:27.977850 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.977826    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99971710-4cf0-4461-810c-bf875fbfcafb" containerName="revoke-keys"
Jul 14 17:59:27.977850 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.977832    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99971710-4cf0-4461-810c-bf875fbfcafb" containerName="revoke-keys"
Jul 14 17:59:27.978002 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.977893    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99971710-4cf0-4461-810c-bf875fbfcafb" containerName="revoke-keys"
Jul 14 17:59:27.978002 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.977903    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4a0a812-228d-441a-9105-56927907dead" containerName="maas-api"
Jul 14 17:59:27.978002 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.977910    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd1d6f30-8634-4d95-89b8-8f9db5fcd7da" containerName="revoke-keys"
Jul 14 17:59:27.982234 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.982198    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 17:59:27.984898 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.984877    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-439ad9d2-serving-cert\""
Jul 14 17:59:27.985005 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.984902    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xxs8g\""
Jul 14 17:59:27.985005 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.984903    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 17:59:27.985528 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.985513    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 17:59:27.992037 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:27.992017    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"]
Jul 14 17:59:28.007402 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.007330    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:28.103799 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.103765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2f8cd94-c992-409d-816f-6a3e385505f4-maas-api-tls\") pod \"maas-api-e2e-ait-439ad9d2-6f78647869-blff5\" (UID: \"f2f8cd94-c992-409d-816f-6a3e385505f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 17:59:28.103799 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.103802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x285\" (UniqueName: \"kubernetes.io/projected/f2f8cd94-c992-409d-816f-6a3e385505f4-kube-api-access-4x285\") pod \"maas-api-e2e-ait-439ad9d2-6f78647869-blff5\" (UID: \"f2f8cd94-c992-409d-816f-6a3e385505f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 17:59:28.104003 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.103863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f8cd94-c992-409d-816f-6a3e385505f4-openshift-service-ca\") pod \"maas-api-e2e-ait-439ad9d2-6f78647869-blff5\" (UID: \"f2f8cd94-c992-409d-816f-6a3e385505f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 17:59:28.107838 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.107809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:28.204639 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.204594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2f8cd94-c992-409d-816f-6a3e385505f4-maas-api-tls\") pod \"maas-api-e2e-ait-439ad9d2-6f78647869-blff5\" (UID: \"f2f8cd94-c992-409d-816f-6a3e385505f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 17:59:28.204639 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.204639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x285\" (UniqueName: \"kubernetes.io/projected/f2f8cd94-c992-409d-816f-6a3e385505f4-kube-api-access-4x285\") pod \"maas-api-e2e-ait-439ad9d2-6f78647869-blff5\" (UID: \"f2f8cd94-c992-409d-816f-6a3e385505f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 17:59:28.204922 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.204670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f8cd94-c992-409d-816f-6a3e385505f4-openshift-service-ca\") pod \"maas-api-e2e-ait-439ad9d2-6f78647869-blff5\" (UID: \"f2f8cd94-c992-409d-816f-6a3e385505f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 17:59:28.204922 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:59:28.204777    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-serving-cert: secret "maas-api-e2e-ait-439ad9d2-serving-cert" not found
Jul 14 17:59:28.204922 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:59:28.204851    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f8cd94-c992-409d-816f-6a3e385505f4-maas-api-tls podName:f2f8cd94-c992-409d-816f-6a3e385505f4 nodeName:}" failed. No retries permitted until 2026-07-14 17:59:28.704830572 +0000 UTC m=+1625.421224253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2f8cd94-c992-409d-816f-6a3e385505f4-maas-api-tls") pod "maas-api-e2e-ait-439ad9d2-6f78647869-blff5" (UID: "f2f8cd94-c992-409d-816f-6a3e385505f4") : secret "maas-api-e2e-ait-439ad9d2-serving-cert" not found
Jul 14 17:59:28.205273 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.205257    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f8cd94-c992-409d-816f-6a3e385505f4-openshift-service-ca\") pod \"maas-api-e2e-ait-439ad9d2-6f78647869-blff5\" (UID: \"f2f8cd94-c992-409d-816f-6a3e385505f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 17:59:28.213986 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.213955    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x285\" (UniqueName: \"kubernetes.io/projected/f2f8cd94-c992-409d-816f-6a3e385505f4-kube-api-access-4x285\") pod \"maas-api-e2e-ait-439ad9d2-6f78647869-blff5\" (UID: \"f2f8cd94-c992-409d-816f-6a3e385505f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 17:59:28.403547 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.403502    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:28.675353 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.675310    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:28.708309 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.708272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2f8cd94-c992-409d-816f-6a3e385505f4-maas-api-tls\") pod \"maas-api-e2e-ait-439ad9d2-6f78647869-blff5\" (UID: \"f2f8cd94-c992-409d-816f-6a3e385505f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 17:59:28.710917 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.710884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2f8cd94-c992-409d-816f-6a3e385505f4-maas-api-tls\") pod \"maas-api-e2e-ait-439ad9d2-6f78647869-blff5\" (UID: \"f2f8cd94-c992-409d-816f-6a3e385505f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 17:59:28.823099 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.823062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:28.893598 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:28.893562    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 17:59:29.039740 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:29.039717    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"]
Jul 14 17:59:29.041365 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:59:29.041338    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f8cd94_c992_409d_816f_6a3e385505f4.slice/crio-5c3566c3b8eca294a2100468ec0ab60574951bab2269bf8a2bf918cf3fe02ee1 WatchSource:0}: Error finding container 5c3566c3b8eca294a2100468ec0ab60574951bab2269bf8a2bf918cf3fe02ee1: Status 404 returned error can't find the container with id 5c3566c3b8eca294a2100468ec0ab60574951bab2269bf8a2bf918cf3fe02ee1
Jul 14 17:59:29.450250 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:29.450194    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5" event={"ID":"f2f8cd94-c992-409d-816f-6a3e385505f4","Type":"ContainerStarted","Data":"5c3566c3b8eca294a2100468ec0ab60574951bab2269bf8a2bf918cf3fe02ee1"}
Jul 14 17:59:30.023362 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:30.023321    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 17:59:30.403966 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:30.403889    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 17:59:32.467723 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:32.467685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5" event={"ID":"f2f8cd94-c992-409d-816f-6a3e385505f4","Type":"ContainerStarted","Data":"0b0484c5a736d9ee994e5fc50a80304586f9ee78c243008587df2003a7a80ec3"}
Jul 14 17:59:32.468242 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:32.467838    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 17:59:32.493468 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:32.493420    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5" podStartSLOduration=2.719637465 podStartE2EDuration="5.493404151s" totalImagesPullingTime="2.773766686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:59:32.491851449 +0000 UTC m=+1629.208245141" watchObservedRunningTime="2026-07-14 17:59:32.493404151 +0000 UTC m=+1629.209797842"
Jul 14 17:59:32.914028 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:32.913953    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd"]
Jul 14 17:59:32.917539 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:32.917523    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd"
Jul 14 17:59:32.920329 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:32.920313    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pd7w9\""
Jul 14 17:59:32.926147 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:32.926122    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd"]
Jul 14 17:59:33.048397 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:33.048351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13ccff7-f557-419e-884e-7bf90bc039c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd\" (UID: \"a13ccff7-f557-419e-884e-7bf90bc039c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd"
Jul 14 17:59:33.149311 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:33.149273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13ccff7-f557-419e-884e-7bf90bc039c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd\" (UID: \"a13ccff7-f557-419e-884e-7bf90bc039c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd"
Jul 14 17:59:33.149874 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:33.149856    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13ccff7-f557-419e-884e-7bf90bc039c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd\" (UID: \"a13ccff7-f557-419e-884e-7bf90bc039c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd"
Jul 14 17:59:33.228258 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:33.228185    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd"
Jul 14 17:59:33.385753 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:33.385721    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd"]
Jul 14 17:59:33.387279 ip-10-0-135-231 kubenswrapper[2578]: W0714 17:59:33.387258    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13ccff7_f557_419e_884e_7bf90bc039c2.slice/crio-f75b0624f3d7ac970f7664464dd1f7c011af6585536545973e5e645e9db21802 WatchSource:0}: Error finding container f75b0624f3d7ac970f7664464dd1f7c011af6585536545973e5e645e9db21802: Status 404 returned error can't find the container with id f75b0624f3d7ac970f7664464dd1f7c011af6585536545973e5e645e9db21802
Jul 14 17:59:33.474500 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:33.474468    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd" event={"ID":"a13ccff7-f557-419e-884e-7bf90bc039c2","Type":"ContainerStarted","Data":"f75b0624f3d7ac970f7664464dd1f7c011af6585536545973e5e645e9db21802"}
Jul 14 17:59:34.481377 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:34.481342    2578 generic.go:356] "Generic (PLEG): container finished" podID="a13ccff7-f557-419e-884e-7bf90bc039c2" containerID="ce51cc905167a1ba69d02cde554b9396f5c61380e9ecd01954851da74e6bd76b" exitCode=7
Jul 14 17:59:34.481773 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:34.481383    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd" event={"ID":"a13ccff7-f557-419e-884e-7bf90bc039c2","Type":"ContainerDied","Data":"ce51cc905167a1ba69d02cde554b9396f5c61380e9ecd01954851da74e6bd76b"}
Jul 14 17:59:34.481773 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:34.481607    2578 scope.go:117] "RemoveContainer" containerID="ce51cc905167a1ba69d02cde554b9396f5c61380e9ecd01954851da74e6bd76b"
Jul 14 17:59:35.489600 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:35.489572    2578 generic.go:356] "Generic (PLEG): container finished" podID="a13ccff7-f557-419e-884e-7bf90bc039c2" containerID="a5ce935aac39021284902a4156b52af4feb20e8b759ee734f1159e14325e459a" exitCode=7
Jul 14 17:59:35.490012 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:35.489666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd" event={"ID":"a13ccff7-f557-419e-884e-7bf90bc039c2","Type":"ContainerDied","Data":"a5ce935aac39021284902a4156b52af4feb20e8b759ee734f1159e14325e459a"}
Jul 14 17:59:35.490012 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:35.489732    2578 scope.go:117] "RemoveContainer" containerID="ce51cc905167a1ba69d02cde554b9396f5c61380e9ecd01954851da74e6bd76b"
Jul 14 17:59:35.490012 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:35.489900    2578 scope.go:117] "RemoveContainer" containerID="a5ce935aac39021284902a4156b52af4feb20e8b759ee734f1159e14325e459a"
Jul 14 17:59:35.490142 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:59:35.490095    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd_odh-ai-gateway-infra(a13ccff7-f557-419e-884e-7bf90bc039c2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd" podUID="a13ccff7-f557-419e-884e-7bf90bc039c2"
Jul 14 17:59:36.497336 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:36.497311    2578 scope.go:117] "RemoveContainer" containerID="a5ce935aac39021284902a4156b52af4feb20e8b759ee734f1159e14325e459a"
Jul 14 17:59:36.497697 ip-10-0-135-231 kubenswrapper[2578]: E0714 17:59:36.497517    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd_odh-ai-gateway-infra(a13ccff7-f557-419e-884e-7bf90bc039c2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd" podUID="a13ccff7-f557-419e-884e-7bf90bc039c2"
Jul 14 17:59:38.479851 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:38.479822    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 17:59:47.777142 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:47.777101    2578 scope.go:117] "RemoveContainer" containerID="a5ce935aac39021284902a4156b52af4feb20e8b759ee734f1159e14325e459a"
Jul 14 17:59:48.570901 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:48.570873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd" event={"ID":"a13ccff7-f557-419e-884e-7bf90bc039c2","Type":"ContainerStarted","Data":"3edca1ddebd4069a4e645cc1deb7ff0ea9e0bd4a43f6ce4f8dcedbb84a7d1825"}
Jul 14 17:59:48.592432 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:48.592384    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd" podStartSLOduration=16.592368851 podStartE2EDuration="16.592368851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:59:48.589649532 +0000 UTC m=+1645.306043225" watchObservedRunningTime="2026-07-14 17:59:48.592368851 +0000 UTC m=+1645.308762542"
Jul 14 17:59:49.578527 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:49.578496    2578 generic.go:356] "Generic (PLEG): container finished" podID="a13ccff7-f557-419e-884e-7bf90bc039c2" containerID="3edca1ddebd4069a4e645cc1deb7ff0ea9e0bd4a43f6ce4f8dcedbb84a7d1825" exitCode=0
Jul 14 17:59:49.578959 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:49.578568    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd" event={"ID":"a13ccff7-f557-419e-884e-7bf90bc039c2","Type":"ContainerDied","Data":"3edca1ddebd4069a4e645cc1deb7ff0ea9e0bd4a43f6ce4f8dcedbb84a7d1825"}
Jul 14 17:59:49.578959 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:49.578622    2578 scope.go:117] "RemoveContainer" containerID="a5ce935aac39021284902a4156b52af4feb20e8b759ee734f1159e14325e459a"
Jul 14 17:59:49.608104 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:49.608072    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd"]
Jul 14 17:59:50.727279 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:50.727250    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd"
Jul 14 17:59:50.793583 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:50.793551    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13ccff7-f557-419e-884e-7bf90bc039c2-maas-api-service-ca\") pod \"a13ccff7-f557-419e-884e-7bf90bc039c2\" (UID: \"a13ccff7-f557-419e-884e-7bf90bc039c2\") "
Jul 14 17:59:50.793940 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:50.793914    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13ccff7-f557-419e-884e-7bf90bc039c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a13ccff7-f557-419e-884e-7bf90bc039c2" (UID: "a13ccff7-f557-419e-884e-7bf90bc039c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:59:50.895016 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:50.894929    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13ccff7-f557-419e-884e-7bf90bc039c2-maas-api-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 17:59:51.593422 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:51.593383    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd" event={"ID":"a13ccff7-f557-419e-884e-7bf90bc039c2","Type":"ContainerDied","Data":"f75b0624f3d7ac970f7664464dd1f7c011af6585536545973e5e645e9db21802"}
Jul 14 17:59:51.593422 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:51.593407    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd"
Jul 14 17:59:51.593422 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:51.593432    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f75b0624f3d7ac970f7664464dd1f7c011af6585536545973e5e645e9db21802"
Jul 14 17:59:51.629509 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:51.629480    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd"]
Jul 14 17:59:51.638089 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:51.638064    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-26zvd"]
Jul 14 17:59:51.783631 ip-10-0-135-231 kubenswrapper[2578]: I0714 17:59:51.783589    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a13ccff7-f557-419e-884e-7bf90bc039c2" path="/var/lib/kubelet/pods/a13ccff7-f557-419e-884e-7bf90bc039c2/volumes"
Jul 14 18:00:00.165963 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.165929    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734200-swcb7"]
Jul 14 18:00:00.166383 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.166365    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a13ccff7-f557-419e-884e-7bf90bc039c2" containerName="revoke-keys"
Jul 14 18:00:00.166424 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.166386    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a13ccff7-f557-419e-884e-7bf90bc039c2" containerName="revoke-keys"
Jul 14 18:00:00.166424 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.166394    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a13ccff7-f557-419e-884e-7bf90bc039c2" containerName="revoke-keys"
Jul 14 18:00:00.166424 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.166401    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a13ccff7-f557-419e-884e-7bf90bc039c2" containerName="revoke-keys"
Jul 14 18:00:00.166530 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.166479    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a13ccff7-f557-419e-884e-7bf90bc039c2" containerName="revoke-keys"
Jul 14 18:00:00.166530 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.166489    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a13ccff7-f557-419e-884e-7bf90bc039c2" containerName="revoke-keys"
Jul 14 18:00:00.166530 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.166494    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a13ccff7-f557-419e-884e-7bf90bc039c2" containerName="revoke-keys"
Jul 14 18:00:00.169552 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.169537    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-439ad9d2-29734200-tz6mb"]
Jul 14 18:00:00.169667 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.169648    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734200-swcb7"
Jul 14 18:00:00.169892 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.169879    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a13ccff7-f557-419e-884e-7bf90bc039c2" containerName="revoke-keys"
Jul 14 18:00:00.169892 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.169893    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a13ccff7-f557-419e-884e-7bf90bc039c2" containerName="revoke-keys"
Jul 14 18:00:00.173018 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.172999    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-439ad9d2-29734200-tz6mb"
Jul 14 18:00:00.173104 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.173065    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pd7w9\""
Jul 14 18:00:00.182070 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.181947    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734200-swcb7"]
Jul 14 18:00:00.182697 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.182233    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734200-swcb7"
Jul 14 18:00:00.187952 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.186454    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-439ad9d2-29734200-tz6mb"]
Jul 14 18:00:00.195749 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.195724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-439ad9d2-29734200-tz6mb"
Jul 14 18:00:00.377487 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.377458    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-439ad9d2-29734200-tz6mb"]
Jul 14 18:00:00.378085 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:00:00.378064    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7c5107_8ea9_4f20_9396_7ad2f5db715a.slice/crio-191945b6e03c13a6782c51665b3ca2a86d48027da1155aac179eaab35852641d WatchSource:0}: Error finding container 191945b6e03c13a6782c51665b3ca2a86d48027da1155aac179eaab35852641d: Status 404 returned error can't find the container with id 191945b6e03c13a6782c51665b3ca2a86d48027da1155aac179eaab35852641d
Jul 14 18:00:00.556756 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.556618    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734200-swcb7"]
Jul 14 18:00:00.559330 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:00:00.559304    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527c26cc_77b2_4188_8a57_be5c367992a6.slice/crio-1f71112d3b5944ce95deff56ff5ec877c3b643a21b1ee7dabcf2864b1e4c367e WatchSource:0}: Error finding container 1f71112d3b5944ce95deff56ff5ec877c3b643a21b1ee7dabcf2864b1e4c367e: Status 404 returned error can't find the container with id 1f71112d3b5944ce95deff56ff5ec877c3b643a21b1ee7dabcf2864b1e4c367e
Jul 14 18:00:00.649909 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.649875    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734200-swcb7" event={"ID":"527c26cc-77b2-4188-8a57-be5c367992a6","Type":"ContainerStarted","Data":"1f71112d3b5944ce95deff56ff5ec877c3b643a21b1ee7dabcf2864b1e4c367e"}
Jul 14 18:00:00.651399 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.651370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-439ad9d2-29734200-tz6mb" event={"ID":"1d7c5107-8ea9-4f20-9396-7ad2f5db715a","Type":"ContainerStarted","Data":"b16292e59d0862e8b15cc839f8b01afe2d9834b21fffc8ea8dac4fc43f633d8e"}
Jul 14 18:00:00.651536 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.651415    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-439ad9d2-29734200-tz6mb" event={"ID":"1d7c5107-8ea9-4f20-9396-7ad2f5db715a","Type":"ContainerStarted","Data":"191945b6e03c13a6782c51665b3ca2a86d48027da1155aac179eaab35852641d"}
Jul 14 18:00:00.670651 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:00.670603    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-439ad9d2-29734200-tz6mb" podStartSLOduration=0.670586649 podStartE2EDuration="670.586649ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:00:00.668805669 +0000 UTC m=+1657.385199360" watchObservedRunningTime="2026-07-14 18:00:00.670586649 +0000 UTC m=+1657.386980342"
Jul 14 18:00:01.658783 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:01.658687    2578 generic.go:356] "Generic (PLEG): container finished" podID="527c26cc-77b2-4188-8a57-be5c367992a6" containerID="92ae31ab84b7b94add2ae40087f2320839f66fee00c1a1e1b2b715f3bf16dade" exitCode=0
Jul 14 18:00:01.658783 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:01.658771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734200-swcb7" event={"ID":"527c26cc-77b2-4188-8a57-be5c367992a6","Type":"ContainerDied","Data":"92ae31ab84b7b94add2ae40087f2320839f66fee00c1a1e1b2b715f3bf16dade"}
Jul 14 18:00:01.660230 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:01.660181    2578 generic.go:356] "Generic (PLEG): container finished" podID="1d7c5107-8ea9-4f20-9396-7ad2f5db715a" containerID="b16292e59d0862e8b15cc839f8b01afe2d9834b21fffc8ea8dac4fc43f633d8e" exitCode=0
Jul 14 18:00:01.660346 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:01.660255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-439ad9d2-29734200-tz6mb" event={"ID":"1d7c5107-8ea9-4f20-9396-7ad2f5db715a","Type":"ContainerDied","Data":"b16292e59d0862e8b15cc839f8b01afe2d9834b21fffc8ea8dac4fc43f633d8e"}
Jul 14 18:00:02.841569 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:02.841542    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-439ad9d2-29734200-tz6mb"
Jul 14 18:00:02.844535 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:02.844517    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734200-swcb7"
Jul 14 18:00:03.673231 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:03.673181    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734200-swcb7"
Jul 14 18:00:03.673456 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:03.673237    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734200-swcb7" event={"ID":"527c26cc-77b2-4188-8a57-be5c367992a6","Type":"ContainerDied","Data":"1f71112d3b5944ce95deff56ff5ec877c3b643a21b1ee7dabcf2864b1e4c367e"}
Jul 14 18:00:03.673456 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:03.673291    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f71112d3b5944ce95deff56ff5ec877c3b643a21b1ee7dabcf2864b1e4c367e"
Jul 14 18:00:03.674437 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:03.674418    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-439ad9d2-29734200-tz6mb"
Jul 14 18:00:03.674571 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:03.674436    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-439ad9d2-29734200-tz6mb" event={"ID":"1d7c5107-8ea9-4f20-9396-7ad2f5db715a","Type":"ContainerDied","Data":"191945b6e03c13a6782c51665b3ca2a86d48027da1155aac179eaab35852641d"}
Jul 14 18:00:03.674571 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:03.674472    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="191945b6e03c13a6782c51665b3ca2a86d48027da1155aac179eaab35852641d"
Jul 14 18:00:22.967550 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:22.967517    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz"]
Jul 14 18:00:22.968029 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:22.967890    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="527c26cc-77b2-4188-8a57-be5c367992a6" containerName="cleanup"
Jul 14 18:00:22.968029 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:22.967901    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="527c26cc-77b2-4188-8a57-be5c367992a6" containerName="cleanup"
Jul 14 18:00:22.968029 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:22.967910    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d7c5107-8ea9-4f20-9396-7ad2f5db715a" containerName="cleanup"
Jul 14 18:00:22.968029 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:22.967915    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d7c5107-8ea9-4f20-9396-7ad2f5db715a" containerName="cleanup"
Jul 14 18:00:22.968029 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:22.968002    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d7c5107-8ea9-4f20-9396-7ad2f5db715a" containerName="cleanup"
Jul 14 18:00:22.968029 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:22.968011    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="527c26cc-77b2-4188-8a57-be5c367992a6" containerName="cleanup"
Jul 14 18:00:22.970978 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:22.970953    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz"
Jul 14 18:00:22.973746 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:22.973727    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pd7w9\""
Jul 14 18:00:22.980334 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:22.980314    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz"]
Jul 14 18:00:23.082801 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:23.082763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932277ae-a14f-41b7-93b1-07fddff76e5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz\" (UID: \"932277ae-a14f-41b7-93b1-07fddff76e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz"
Jul 14 18:00:23.183265 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:23.183223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932277ae-a14f-41b7-93b1-07fddff76e5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz\" (UID: \"932277ae-a14f-41b7-93b1-07fddff76e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz"
Jul 14 18:00:23.183781 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:23.183760    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932277ae-a14f-41b7-93b1-07fddff76e5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz\" (UID: \"932277ae-a14f-41b7-93b1-07fddff76e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz"
Jul 14 18:00:23.283419 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:23.283326    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz"
Jul 14 18:00:23.427106 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:23.427079    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz"]
Jul 14 18:00:23.428776 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:00:23.428741    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932277ae_a14f_41b7_93b1_07fddff76e5b.slice/crio-2128f5bcd05069595e0e2a9528c984926b9d15857b612d094e3bb1ec763127d8 WatchSource:0}: Error finding container 2128f5bcd05069595e0e2a9528c984926b9d15857b612d094e3bb1ec763127d8: Status 404 returned error can't find the container with id 2128f5bcd05069595e0e2a9528c984926b9d15857b612d094e3bb1ec763127d8
Jul 14 18:00:23.792898 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:23.792861    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz" event={"ID":"932277ae-a14f-41b7-93b1-07fddff76e5b","Type":"ContainerStarted","Data":"856d25ad71e1fb7a52504d58d622ee08e1d6813aae5f3b45b1ecf6e23a3a1744"}
Jul 14 18:00:23.793051 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:23.792917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz" event={"ID":"932277ae-a14f-41b7-93b1-07fddff76e5b","Type":"ContainerStarted","Data":"2128f5bcd05069595e0e2a9528c984926b9d15857b612d094e3bb1ec763127d8"}
Jul 14 18:00:23.822244 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:23.822177    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz" podStartSLOduration=1.8221619549999999 podStartE2EDuration="1.822161955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:00:23.818603993 +0000 UTC m=+1680.534997685" watchObservedRunningTime="2026-07-14 18:00:23.822161955 +0000 UTC m=+1680.538555671"
Jul 14 18:00:24.800285 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:24.800253    2578 generic.go:356] "Generic (PLEG): container finished" podID="932277ae-a14f-41b7-93b1-07fddff76e5b" containerID="856d25ad71e1fb7a52504d58d622ee08e1d6813aae5f3b45b1ecf6e23a3a1744" exitCode=0
Jul 14 18:00:24.800657 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:24.800347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz" event={"ID":"932277ae-a14f-41b7-93b1-07fddff76e5b","Type":"ContainerDied","Data":"856d25ad71e1fb7a52504d58d622ee08e1d6813aae5f3b45b1ecf6e23a3a1744"}
Jul 14 18:00:25.605431 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:25.605392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:25.935144 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:25.935119    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz"
Jul 14 18:00:26.110516 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:26.110428    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932277ae-a14f-41b7-93b1-07fddff76e5b-maas-api-service-ca\") pod \"932277ae-a14f-41b7-93b1-07fddff76e5b\" (UID: \"932277ae-a14f-41b7-93b1-07fddff76e5b\") "
Jul 14 18:00:26.110828 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:26.110805    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932277ae-a14f-41b7-93b1-07fddff76e5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "932277ae-a14f-41b7-93b1-07fddff76e5b" (UID: "932277ae-a14f-41b7-93b1-07fddff76e5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:00:26.211290 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:26.211258    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932277ae-a14f-41b7-93b1-07fddff76e5b-maas-api-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:00:26.811759 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:26.811719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz" event={"ID":"932277ae-a14f-41b7-93b1-07fddff76e5b","Type":"ContainerDied","Data":"2128f5bcd05069595e0e2a9528c984926b9d15857b612d094e3bb1ec763127d8"}
Jul 14 18:00:26.811759 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:26.811747    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz"
Jul 14 18:00:26.811968 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:26.811778    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2128f5bcd05069595e0e2a9528c984926b9d15857b612d094e3bb1ec763127d8"
Jul 14 18:00:27.965419 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:27.965382    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:33.529804 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:33.529771    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:33.626993 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:33.626954    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:33.787105 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:33.787036    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"]
Jul 14 18:00:33.787264 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:33.787243    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5" podUID="f2f8cd94-c992-409d-816f-6a3e385505f4" containerName="maas-api" containerID="cri-o://0b0484c5a736d9ee994e5fc50a80304586f9ee78c243008587df2003a7a80ec3" gracePeriod=30
Jul 14 18:00:33.812821 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:33.812794    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:33.814056 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:33.814019    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-439ad9d2-29734200-tz6mb"]
Jul 14 18:00:33.816466 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:33.816441    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-439ad9d2-29734200-tz6mb"]
Jul 14 18:00:34.035745 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.035715    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 18:00:34.085770 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.085686    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f8cd94-c992-409d-816f-6a3e385505f4-openshift-service-ca\") pod \"f2f8cd94-c992-409d-816f-6a3e385505f4\" (UID: \"f2f8cd94-c992-409d-816f-6a3e385505f4\") "
Jul 14 18:00:34.085770 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.085729    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x285\" (UniqueName: \"kubernetes.io/projected/f2f8cd94-c992-409d-816f-6a3e385505f4-kube-api-access-4x285\") pod \"f2f8cd94-c992-409d-816f-6a3e385505f4\" (UID: \"f2f8cd94-c992-409d-816f-6a3e385505f4\") "
Jul 14 18:00:34.085954 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.085875    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2f8cd94-c992-409d-816f-6a3e385505f4-maas-api-tls\") pod \"f2f8cd94-c992-409d-816f-6a3e385505f4\" (UID: \"f2f8cd94-c992-409d-816f-6a3e385505f4\") "
Jul 14 18:00:34.086132 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.086102    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f8cd94-c992-409d-816f-6a3e385505f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f2f8cd94-c992-409d-816f-6a3e385505f4" (UID: "f2f8cd94-c992-409d-816f-6a3e385505f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:00:34.086297 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.086189    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f8cd94-c992-409d-816f-6a3e385505f4-openshift-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:00:34.087949 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.087919    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2f8cd94-c992-409d-816f-6a3e385505f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f2f8cd94-c992-409d-816f-6a3e385505f4" (UID: "f2f8cd94-c992-409d-816f-6a3e385505f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:00:34.088036 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.087984    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f8cd94-c992-409d-816f-6a3e385505f4-kube-api-access-4x285" (OuterVolumeSpecName: "kube-api-access-4x285") pod "f2f8cd94-c992-409d-816f-6a3e385505f4" (UID: "f2f8cd94-c992-409d-816f-6a3e385505f4"). InnerVolumeSpecName "kube-api-access-4x285". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:00:34.186718 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.186685    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x285\" (UniqueName: \"kubernetes.io/projected/f2f8cd94-c992-409d-816f-6a3e385505f4-kube-api-access-4x285\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:00:34.186718 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.186718    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2f8cd94-c992-409d-816f-6a3e385505f4-maas-api-tls\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:00:34.858276 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.858239    2578 generic.go:356] "Generic (PLEG): container finished" podID="f2f8cd94-c992-409d-816f-6a3e385505f4" containerID="0b0484c5a736d9ee994e5fc50a80304586f9ee78c243008587df2003a7a80ec3" exitCode=0
Jul 14 18:00:34.858671 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.858298    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"
Jul 14 18:00:34.858671 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.858329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5" event={"ID":"f2f8cd94-c992-409d-816f-6a3e385505f4","Type":"ContainerDied","Data":"0b0484c5a736d9ee994e5fc50a80304586f9ee78c243008587df2003a7a80ec3"}
Jul 14 18:00:34.858671 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.858386    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5" event={"ID":"f2f8cd94-c992-409d-816f-6a3e385505f4","Type":"ContainerDied","Data":"5c3566c3b8eca294a2100468ec0ab60574951bab2269bf8a2bf918cf3fe02ee1"}
Jul 14 18:00:34.858671 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.858408    2578 scope.go:117] "RemoveContainer" containerID="0b0484c5a736d9ee994e5fc50a80304586f9ee78c243008587df2003a7a80ec3"
Jul 14 18:00:34.868351 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.868332    2578 scope.go:117] "RemoveContainer" containerID="0b0484c5a736d9ee994e5fc50a80304586f9ee78c243008587df2003a7a80ec3"
Jul 14 18:00:34.868613 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:00:34.868595    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0484c5a736d9ee994e5fc50a80304586f9ee78c243008587df2003a7a80ec3\": container with ID starting with 0b0484c5a736d9ee994e5fc50a80304586f9ee78c243008587df2003a7a80ec3 not found: ID does not exist" containerID="0b0484c5a736d9ee994e5fc50a80304586f9ee78c243008587df2003a7a80ec3"
Jul 14 18:00:34.868672 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.868624    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0484c5a736d9ee994e5fc50a80304586f9ee78c243008587df2003a7a80ec3"} err="failed to get container status \"0b0484c5a736d9ee994e5fc50a80304586f9ee78c243008587df2003a7a80ec3\": rpc error: code = NotFound desc = could not find container \"0b0484c5a736d9ee994e5fc50a80304586f9ee78c243008587df2003a7a80ec3\": container with ID starting with 0b0484c5a736d9ee994e5fc50a80304586f9ee78c243008587df2003a7a80ec3 not found: ID does not exist"
Jul 14 18:00:34.883614 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.883588    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"]
Jul 14 18:00:34.890515 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:34.890494    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-439ad9d2-6f78647869-blff5"]
Jul 14 18:00:35.783386 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:35.783352    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d7c5107-8ea9-4f20-9396-7ad2f5db715a" path="/var/lib/kubelet/pods/1d7c5107-8ea9-4f20-9396-7ad2f5db715a/volumes"
Jul 14 18:00:35.783688 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:35.783676    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2f8cd94-c992-409d-816f-6a3e385505f4" path="/var/lib/kubelet/pods/f2f8cd94-c992-409d-816f-6a3e385505f4/volumes"
Jul 14 18:00:36.794261 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:36.794230    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:00:38.069123 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:38.069092    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:44.115002 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:44.114920    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:44.221248 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:44.221187    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:49.097035 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:49.096997    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:49.402425 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:49.402387    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:50.570366 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.570333    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:50.688488 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.688447    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"]
Jul 14 18:00:50.688872 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.688858    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f8cd94-c992-409d-816f-6a3e385505f4" containerName="maas-api"
Jul 14 18:00:50.688952 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.688876    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f8cd94-c992-409d-816f-6a3e385505f4" containerName="maas-api"
Jul 14 18:00:50.688952 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.688907    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="932277ae-a14f-41b7-93b1-07fddff76e5b" containerName="revoke-keys"
Jul 14 18:00:50.688952 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.688913    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="932277ae-a14f-41b7-93b1-07fddff76e5b" containerName="revoke-keys"
Jul 14 18:00:50.689092 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.688992    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2f8cd94-c992-409d-816f-6a3e385505f4" containerName="maas-api"
Jul 14 18:00:50.689092 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.689005    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="932277ae-a14f-41b7-93b1-07fddff76e5b" containerName="revoke-keys"
Jul 14 18:00:50.690556 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.690532    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:50.693516 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.693501    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:00:50.695995 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.695976    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-68125063-serving-cert\""
Jul 14 18:00:50.696120 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.696104    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 18:00:50.696807 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.696790    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xxs8g\""
Jul 14 18:00:50.696869 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.696795    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 18:00:50.706563 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.706543    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"]
Jul 14 18:00:50.731318 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.731296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-maas-api-tls\") pod \"maas-api-e2e-derive-68125063-dfd49494c-pjxwx\" (UID: \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:00:50.731431 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.731326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24vc9\" (UniqueName: \"kubernetes.io/projected/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-kube-api-access-24vc9\") pod \"maas-api-e2e-derive-68125063-dfd49494c-pjxwx\" (UID: \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:00:50.731431 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.731354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-openshift-service-ca\") pod \"maas-api-e2e-derive-68125063-dfd49494c-pjxwx\" (UID: \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:00:50.832604 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.832528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-maas-api-tls\") pod \"maas-api-e2e-derive-68125063-dfd49494c-pjxwx\" (UID: \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:00:50.832604 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.832561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24vc9\" (UniqueName: \"kubernetes.io/projected/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-kube-api-access-24vc9\") pod \"maas-api-e2e-derive-68125063-dfd49494c-pjxwx\" (UID: \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:00:50.832604 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.832591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-openshift-service-ca\") pod \"maas-api-e2e-derive-68125063-dfd49494c-pjxwx\" (UID: \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:00:50.832860 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:00:50.832679    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-68125063-serving-cert: secret "maas-api-e2e-derive-68125063-serving-cert" not found
Jul 14 18:00:50.832860 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:00:50.832755    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-maas-api-tls podName:49ef0f5d-0d5d-4b47-8854-2d08d9471cc5 nodeName:}" failed. No retries permitted until 2026-07-14 18:00:51.33273857 +0000 UTC m=+1708.049132241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-maas-api-tls") pod "maas-api-e2e-derive-68125063-dfd49494c-pjxwx" (UID: "49ef0f5d-0d5d-4b47-8854-2d08d9471cc5") : secret "maas-api-e2e-derive-68125063-serving-cert" not found
Jul 14 18:00:50.833134 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.833117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-openshift-service-ca\") pod \"maas-api-e2e-derive-68125063-dfd49494c-pjxwx\" (UID: \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:00:50.844592 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.844567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24vc9\" (UniqueName: \"kubernetes.io/projected/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-kube-api-access-24vc9\") pod \"maas-api-e2e-derive-68125063-dfd49494c-pjxwx\" (UID: \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:00:50.921556 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:50.921519    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:51.269066 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:51.269028    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:51.337263 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:51.337229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-maas-api-tls\") pod \"maas-api-e2e-derive-68125063-dfd49494c-pjxwx\" (UID: \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:00:51.339753 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:51.339734    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-maas-api-tls\") pod \"maas-api-e2e-derive-68125063-dfd49494c-pjxwx\" (UID: \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:00:51.499243 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:51.499185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:51.600386 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:51.600308    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:51.603311 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:51.603291    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:00:51.673386 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:51.673323    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:51.741365 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:51.741230    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"]
Jul 14 18:00:51.743836 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:00:51.743810    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ef0f5d_0d5d_4b47_8854_2d08d9471cc5.slice/crio-bac75c1956e07f34987c52dc76c6ee46cda9a159b0da4dc00915f4530bdc24ba WatchSource:0}: Error finding container bac75c1956e07f34987c52dc76c6ee46cda9a159b0da4dc00915f4530bdc24ba: Status 404 returned error can't find the container with id bac75c1956e07f34987c52dc76c6ee46cda9a159b0da4dc00915f4530bdc24ba
Jul 14 18:00:51.955456 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:51.955425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx" event={"ID":"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5","Type":"ContainerStarted","Data":"bac75c1956e07f34987c52dc76c6ee46cda9a159b0da4dc00915f4530bdc24ba"}
Jul 14 18:00:52.961819 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:52.961783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx" event={"ID":"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5","Type":"ContainerStarted","Data":"97ab768ab7c7568c58b9e5f0cb50cea2c7d2cc04574a03ed48c1f5edc67d0b11"}
Jul 14 18:00:52.962229 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:52.961864    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:00:52.984056 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:52.984000    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx" podStartSLOduration=2.188135996 podStartE2EDuration="2.983980257s" totalImagesPullingTime="795.844261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:00:52.983370401 +0000 UTC m=+1709.699764113" watchObservedRunningTime="2026-07-14 18:00:52.983980257 +0000 UTC m=+1709.700373949"
Jul 14 18:00:54.879334 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:54.879292    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q"]
Jul 14 18:00:54.882868 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:54.882852    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q"
Jul 14 18:00:54.885097 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:54.885074    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pd7w9\""
Jul 14 18:00:54.891909 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:54.891872    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q"]
Jul 14 18:00:54.972714 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:54.972685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34a631b-690b-4a04-a4c4-f64150565e99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-68125063-j5r7q\" (UID: \"a34a631b-690b-4a04-a4c4-f64150565e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q"
Jul 14 18:00:55.073495 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:55.073458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34a631b-690b-4a04-a4c4-f64150565e99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-68125063-j5r7q\" (UID: \"a34a631b-690b-4a04-a4c4-f64150565e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q"
Jul 14 18:00:55.074103 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:55.074083    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34a631b-690b-4a04-a4c4-f64150565e99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-68125063-j5r7q\" (UID: \"a34a631b-690b-4a04-a4c4-f64150565e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q"
Jul 14 18:00:55.194900 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:55.194861    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q"
Jul 14 18:00:55.335195 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:55.335168    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q"]
Jul 14 18:00:55.336729 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:00:55.336695    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34a631b_690b_4a04_a4c4_f64150565e99.slice/crio-e5d3a2bc79358defb3a4f505886cfbfc41bf6f8eda19a09f7512ec3d5ce3e5e9 WatchSource:0}: Error finding container e5d3a2bc79358defb3a4f505886cfbfc41bf6f8eda19a09f7512ec3d5ce3e5e9: Status 404 returned error can't find the container with id e5d3a2bc79358defb3a4f505886cfbfc41bf6f8eda19a09f7512ec3d5ce3e5e9
Jul 14 18:00:55.980704 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:55.980659    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q" event={"ID":"a34a631b-690b-4a04-a4c4-f64150565e99","Type":"ContainerStarted","Data":"712b605b13503b979a8d8f71401d12f91b76b9fb23e2aa94955a8a9618a52360"}
Jul 14 18:00:55.981078 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:55.980711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q" event={"ID":"a34a631b-690b-4a04-a4c4-f64150565e99","Type":"ContainerStarted","Data":"e5d3a2bc79358defb3a4f505886cfbfc41bf6f8eda19a09f7512ec3d5ce3e5e9"}
Jul 14 18:00:55.999999 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:55.999953    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q" podStartSLOduration=1.999938472 podStartE2EDuration="1.999938472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:00:55.997311735 +0000 UTC m=+1712.713705439" watchObservedRunningTime="2026-07-14 18:00:55.999938472 +0000 UTC m=+1712.716332164"
Jul 14 18:00:56.317335 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:56.317304    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:56.445275 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:56.445225    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:00:56.988658 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:56.988620    2578 generic.go:356] "Generic (PLEG): container finished" podID="a34a631b-690b-4a04-a4c4-f64150565e99" containerID="712b605b13503b979a8d8f71401d12f91b76b9fb23e2aa94955a8a9618a52360" exitCode=7
Jul 14 18:00:56.989082 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:56.988665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q" event={"ID":"a34a631b-690b-4a04-a4c4-f64150565e99","Type":"ContainerDied","Data":"712b605b13503b979a8d8f71401d12f91b76b9fb23e2aa94955a8a9618a52360"}
Jul 14 18:00:56.989082 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:56.988918    2578 scope.go:117] "RemoveContainer" containerID="712b605b13503b979a8d8f71401d12f91b76b9fb23e2aa94955a8a9618a52360"
Jul 14 18:00:57.701398 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:57.701366    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:00:57.996782 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:57.996753    2578 generic.go:356] "Generic (PLEG): container finished" podID="a34a631b-690b-4a04-a4c4-f64150565e99" containerID="111fd72e81cdd040ebc8b59a9f8bd6c819e2f2fe4a865c2140a4fd7655c204ae" exitCode=7
Jul 14 18:00:57.997265 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:57.996841    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q" event={"ID":"a34a631b-690b-4a04-a4c4-f64150565e99","Type":"ContainerDied","Data":"111fd72e81cdd040ebc8b59a9f8bd6c819e2f2fe4a865c2140a4fd7655c204ae"}
Jul 14 18:00:57.997265 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:57.996905    2578 scope.go:117] "RemoveContainer" containerID="712b605b13503b979a8d8f71401d12f91b76b9fb23e2aa94955a8a9618a52360"
Jul 14 18:00:57.997265 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:57.997122    2578 scope.go:117] "RemoveContainer" containerID="111fd72e81cdd040ebc8b59a9f8bd6c819e2f2fe4a865c2140a4fd7655c204ae"
Jul 14 18:00:57.997419 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:00:57.997396    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-68125063-j5r7q_odh-ai-gateway-infra(a34a631b-690b-4a04-a4c4-f64150565e99)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q" podUID="a34a631b-690b-4a04-a4c4-f64150565e99"
Jul 14 18:00:58.974320 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:58.974292    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:00:59.004793 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:59.004754    2578 scope.go:117] "RemoveContainer" containerID="111fd72e81cdd040ebc8b59a9f8bd6c819e2f2fe4a865c2140a4fd7655c204ae"
Jul 14 18:00:59.005271 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:00:59.005047    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-68125063-j5r7q_odh-ai-gateway-infra(a34a631b-690b-4a04-a4c4-f64150565e99)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q" podUID="a34a631b-690b-4a04-a4c4-f64150565e99"
Jul 14 18:00:59.950836 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:00:59.950805    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:01:11.777627 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:11.777595    2578 scope.go:117] "RemoveContainer" containerID="111fd72e81cdd040ebc8b59a9f8bd6c819e2f2fe4a865c2140a4fd7655c204ae"
Jul 14 18:01:13.089122 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:13.089085    2578 generic.go:356] "Generic (PLEG): container finished" podID="a34a631b-690b-4a04-a4c4-f64150565e99" containerID="4a67520b7796daec3a31578901327e88ad15d53703a754178d13ad62a97dfb13" exitCode=0
Jul 14 18:01:13.089526 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:13.089155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q" event={"ID":"a34a631b-690b-4a04-a4c4-f64150565e99","Type":"ContainerDied","Data":"4a67520b7796daec3a31578901327e88ad15d53703a754178d13ad62a97dfb13"}
Jul 14 18:01:13.089526 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:13.089236    2578 scope.go:117] "RemoveContainer" containerID="111fd72e81cdd040ebc8b59a9f8bd6c819e2f2fe4a865c2140a4fd7655c204ae"
Jul 14 18:01:14.129188 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:14.129155    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q"]
Jul 14 18:01:14.235750 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:14.235721    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q"
Jul 14 18:01:14.345901 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:14.345871    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34a631b-690b-4a04-a4c4-f64150565e99-maas-api-service-ca\") pod \"a34a631b-690b-4a04-a4c4-f64150565e99\" (UID: \"a34a631b-690b-4a04-a4c4-f64150565e99\") "
Jul 14 18:01:14.346302 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:14.346278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34a631b-690b-4a04-a4c4-f64150565e99-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a34a631b-690b-4a04-a4c4-f64150565e99" (UID: "a34a631b-690b-4a04-a4c4-f64150565e99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:01:14.446670 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:14.446637    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34a631b-690b-4a04-a4c4-f64150565e99-maas-api-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:01:15.103639 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:15.103611    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q"
Jul 14 18:01:15.103797 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:15.103611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q" event={"ID":"a34a631b-690b-4a04-a4c4-f64150565e99","Type":"ContainerDied","Data":"e5d3a2bc79358defb3a4f505886cfbfc41bf6f8eda19a09f7512ec3d5ce3e5e9"}
Jul 14 18:01:15.103797 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:15.103740    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5d3a2bc79358defb3a4f505886cfbfc41bf6f8eda19a09f7512ec3d5ce3e5e9"
Jul 14 18:01:15.140698 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:15.140667    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q"]
Jul 14 18:01:15.149529 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:15.149504    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-j5r7q"]
Jul 14 18:01:15.783342 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:15.783311    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a34a631b-690b-4a04-a4c4-f64150565e99" path="/var/lib/kubelet/pods/a34a631b-690b-4a04-a4c4-f64150565e99/volumes"
Jul 14 18:01:54.940523 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:54.940486    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-kznjt"]
Jul 14 18:01:54.941002 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:54.940898    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34a631b-690b-4a04-a4c4-f64150565e99" containerName="revoke-keys"
Jul 14 18:01:54.941002 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:54.940911    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34a631b-690b-4a04-a4c4-f64150565e99" containerName="revoke-keys"
Jul 14 18:01:54.941002 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:54.940940    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34a631b-690b-4a04-a4c4-f64150565e99" containerName="revoke-keys"
Jul 14 18:01:54.941002 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:54.940946    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34a631b-690b-4a04-a4c4-f64150565e99" containerName="revoke-keys"
Jul 14 18:01:54.941002 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:54.941001    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34a631b-690b-4a04-a4c4-f64150565e99" containerName="revoke-keys"
Jul 14 18:01:54.941189 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:54.941009    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34a631b-690b-4a04-a4c4-f64150565e99" containerName="revoke-keys"
Jul 14 18:01:54.944066 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:54.944051    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-kznjt"
Jul 14 18:01:54.946708 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:54.946689    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pd7w9\""
Jul 14 18:01:54.953189 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:54.953165    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-kznjt"]
Jul 14 18:01:54.992897 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:54.992870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7013419-68fa-4ef1-b415-0c501e86a769-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-68125063-kznjt\" (UID: \"f7013419-68fa-4ef1-b415-0c501e86a769\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-kznjt"
Jul 14 18:01:55.094284 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:55.094249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7013419-68fa-4ef1-b415-0c501e86a769-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-68125063-kznjt\" (UID: \"f7013419-68fa-4ef1-b415-0c501e86a769\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-kznjt"
Jul 14 18:01:55.094916 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:55.094889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7013419-68fa-4ef1-b415-0c501e86a769-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-68125063-kznjt\" (UID: \"f7013419-68fa-4ef1-b415-0c501e86a769\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-kznjt"
Jul 14 18:01:55.256162 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:55.256079    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-kznjt"
Jul 14 18:01:55.398896 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:01:55.398865    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7013419_68fa_4ef1_b415_0c501e86a769.slice/crio-ba41bd2529887ae43fbddbab862fc0045d006ebd938ffc063bd3458d3ad3ae93 WatchSource:0}: Error finding container ba41bd2529887ae43fbddbab862fc0045d006ebd938ffc063bd3458d3ad3ae93: Status 404 returned error can't find the container with id ba41bd2529887ae43fbddbab862fc0045d006ebd938ffc063bd3458d3ad3ae93
Jul 14 18:01:55.399026 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:55.398914    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-kznjt"]
Jul 14 18:01:56.340363 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:56.340265    2578 generic.go:356] "Generic (PLEG): container finished" podID="f7013419-68fa-4ef1-b415-0c501e86a769" containerID="9e6c7c784ec80a1080656f5740b027582f4c573d272121c4d73e181903496110" exitCode=0
Jul 14 18:01:56.340746 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:56.340356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-kznjt" event={"ID":"f7013419-68fa-4ef1-b415-0c501e86a769","Type":"ContainerDied","Data":"9e6c7c784ec80a1080656f5740b027582f4c573d272121c4d73e181903496110"}
Jul 14 18:01:56.340746 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:56.340416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-kznjt" event={"ID":"f7013419-68fa-4ef1-b415-0c501e86a769","Type":"ContainerStarted","Data":"ba41bd2529887ae43fbddbab862fc0045d006ebd938ffc063bd3458d3ad3ae93"}
Jul 14 18:01:57.493948 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:57.493920    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-kznjt"
Jul 14 18:01:57.615259 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:57.615137    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7013419-68fa-4ef1-b415-0c501e86a769-maas-api-service-ca\") pod \"f7013419-68fa-4ef1-b415-0c501e86a769\" (UID: \"f7013419-68fa-4ef1-b415-0c501e86a769\") "
Jul 14 18:01:57.615544 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:57.615519    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7013419-68fa-4ef1-b415-0c501e86a769-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7013419-68fa-4ef1-b415-0c501e86a769" (UID: "f7013419-68fa-4ef1-b415-0c501e86a769"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:01:57.716114 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:57.716083    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7013419-68fa-4ef1-b415-0c501e86a769-maas-api-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:01:58.353159 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:58.353128    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-kznjt"
Jul 14 18:01:58.353159 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:58.353135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68125063-kznjt" event={"ID":"f7013419-68fa-4ef1-b415-0c501e86a769","Type":"ContainerDied","Data":"ba41bd2529887ae43fbddbab862fc0045d006ebd938ffc063bd3458d3ad3ae93"}
Jul 14 18:01:58.353393 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:01:58.353230    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba41bd2529887ae43fbddbab862fc0045d006ebd938ffc063bd3458d3ad3ae93"
Jul 14 18:02:05.604974 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:05.604933    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"]
Jul 14 18:02:05.605453 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:05.605260    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx" podUID="49ef0f5d-0d5d-4b47-8854-2d08d9471cc5" containerName="maas-api" containerID="cri-o://97ab768ab7c7568c58b9e5f0cb50cea2c7d2cc04574a03ed48c1f5edc67d0b11" gracePeriod=30
Jul 14 18:02:05.636089 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:05.636062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:02:05.858740 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:05.858682    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:02:05.990975 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:05.990941    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-openshift-service-ca\") pod \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\" (UID: \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\") "
Jul 14 18:02:05.990975 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:05.990979    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-maas-api-tls\") pod \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\" (UID: \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\") "
Jul 14 18:02:05.991265 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:05.991048    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24vc9\" (UniqueName: \"kubernetes.io/projected/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-kube-api-access-24vc9\") pod \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\" (UID: \"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5\") "
Jul 14 18:02:05.991407 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:05.991382    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "49ef0f5d-0d5d-4b47-8854-2d08d9471cc5" (UID: "49ef0f5d-0d5d-4b47-8854-2d08d9471cc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:02:05.993338 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:05.993310    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "49ef0f5d-0d5d-4b47-8854-2d08d9471cc5" (UID: "49ef0f5d-0d5d-4b47-8854-2d08d9471cc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:02:05.993482 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:05.993466    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-kube-api-access-24vc9" (OuterVolumeSpecName: "kube-api-access-24vc9") pod "49ef0f5d-0d5d-4b47-8854-2d08d9471cc5" (UID: "49ef0f5d-0d5d-4b47-8854-2d08d9471cc5"). InnerVolumeSpecName "kube-api-access-24vc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:02:06.092663 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:06.092627    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24vc9\" (UniqueName: \"kubernetes.io/projected/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-kube-api-access-24vc9\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:02:06.092663 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:06.092659    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-openshift-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:02:06.092663 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:06.092669    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5-maas-api-tls\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:02:06.403389 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:06.403354    2578 generic.go:356] "Generic (PLEG): container finished" podID="49ef0f5d-0d5d-4b47-8854-2d08d9471cc5" containerID="97ab768ab7c7568c58b9e5f0cb50cea2c7d2cc04574a03ed48c1f5edc67d0b11" exitCode=0
Jul 14 18:02:06.403644 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:06.403403    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"
Jul 14 18:02:06.403644 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:06.403441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx" event={"ID":"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5","Type":"ContainerDied","Data":"97ab768ab7c7568c58b9e5f0cb50cea2c7d2cc04574a03ed48c1f5edc67d0b11"}
Jul 14 18:02:06.403644 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:06.403495    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx" event={"ID":"49ef0f5d-0d5d-4b47-8854-2d08d9471cc5","Type":"ContainerDied","Data":"bac75c1956e07f34987c52dc76c6ee46cda9a159b0da4dc00915f4530bdc24ba"}
Jul 14 18:02:06.403644 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:06.403519    2578 scope.go:117] "RemoveContainer" containerID="97ab768ab7c7568c58b9e5f0cb50cea2c7d2cc04574a03ed48c1f5edc67d0b11"
Jul 14 18:02:06.413662 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:06.413644    2578 scope.go:117] "RemoveContainer" containerID="97ab768ab7c7568c58b9e5f0cb50cea2c7d2cc04574a03ed48c1f5edc67d0b11"
Jul 14 18:02:06.413927 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:02:06.413907    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ab768ab7c7568c58b9e5f0cb50cea2c7d2cc04574a03ed48c1f5edc67d0b11\": container with ID starting with 97ab768ab7c7568c58b9e5f0cb50cea2c7d2cc04574a03ed48c1f5edc67d0b11 not found: ID does not exist" containerID="97ab768ab7c7568c58b9e5f0cb50cea2c7d2cc04574a03ed48c1f5edc67d0b11"
Jul 14 18:02:06.414008 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:06.413941    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ab768ab7c7568c58b9e5f0cb50cea2c7d2cc04574a03ed48c1f5edc67d0b11"} err="failed to get container status \"97ab768ab7c7568c58b9e5f0cb50cea2c7d2cc04574a03ed48c1f5edc67d0b11\": rpc error: code = NotFound desc = could not find container \"97ab768ab7c7568c58b9e5f0cb50cea2c7d2cc04574a03ed48c1f5edc67d0b11\": container with ID starting with 97ab768ab7c7568c58b9e5f0cb50cea2c7d2cc04574a03ed48c1f5edc67d0b11 not found: ID does not exist"
Jul 14 18:02:06.426558 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:06.426533    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"]
Jul 14 18:02:06.431581 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:06.431561    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68125063-dfd49494c-pjxwx"]
Jul 14 18:02:07.783879 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:07.783839    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49ef0f5d-0d5d-4b47-8854-2d08d9471cc5" path="/var/lib/kubelet/pods/49ef0f5d-0d5d-4b47-8854-2d08d9471cc5/volumes"
Jul 14 18:02:08.202408 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:08.196848    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:02:10.040899 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:10.040864    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:02:16.544142 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:16.544056    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:02:16.644823 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:16.644789    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:02:20.290979 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:20.290945    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:02:20.849100 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:20.849065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:02:20.947223 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:20.947185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:02:21.052457 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:21.052412    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:02:22.572500 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:22.572465    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:02:24.299464 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:24.299356    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 18:02:24.310144 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:24.302090    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 18:02:27.226751 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:27.226714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:02:28.967250 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:28.967200    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:02:29.068104 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:29.068073    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:02:29.173790 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:29.173754    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:02:29.274542 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:29.274462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:02:29.719806 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:29.719778    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:02:55.680506 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:02:55.680470    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:02.686391 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:02.686354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:02.756394 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:02.756347    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:02.859470 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:02.859429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:02.974923 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:02.974826    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:06.208660 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:06.208628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:18.866608 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:18.866573    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:25.662629 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:25.662595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:25.766244 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:25.766184    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:25.880405 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:25.880375    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:31.075997 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:31.075967    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:36.424646 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:36.424609    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:37.217827 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:37.217793    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:44.076308 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:44.076198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:44.188282 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:44.188247    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:44.288098 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:44.288070    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:44.394308 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:44.394220    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:45.167782 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:45.167753    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:45.268446 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:45.268409    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:45.376632 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:45.376595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:47.035622 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:47.035590    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:47.529248 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:47.529201    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:03:53.953858 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:53.953822    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:54.060770 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:54.060737    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:54.653722 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:54.653688    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:03:56.407921 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:03:56.407884    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:04:02.065169 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:02.065132    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:02.166752 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:02.166717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:02.321328 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:02.321243    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:02.745752 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:02.745719    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:03.004014 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:03.003938    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:05.788565 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.788530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:05.883328 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.883294    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"]
Jul 14 18:04:05.883738 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.883719    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7013419-68fa-4ef1-b415-0c501e86a769" containerName="revoke-keys"
Jul 14 18:04:05.883828 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.883739    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7013419-68fa-4ef1-b415-0c501e86a769" containerName="revoke-keys"
Jul 14 18:04:05.883828 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.883763    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34a631b-690b-4a04-a4c4-f64150565e99" containerName="revoke-keys"
Jul 14 18:04:05.883828 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.883772    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34a631b-690b-4a04-a4c4-f64150565e99" containerName="revoke-keys"
Jul 14 18:04:05.883828 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.883786    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ef0f5d-0d5d-4b47-8854-2d08d9471cc5" containerName="maas-api"
Jul 14 18:04:05.883828 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.883795    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ef0f5d-0d5d-4b47-8854-2d08d9471cc5" containerName="maas-api"
Jul 14 18:04:05.884082 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.883889    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34a631b-690b-4a04-a4c4-f64150565e99" containerName="revoke-keys"
Jul 14 18:04:05.884082 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.883901    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7013419-68fa-4ef1-b415-0c501e86a769" containerName="revoke-keys"
Jul 14 18:04:05.884082 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.883912    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ef0f5d-0d5d-4b47-8854-2d08d9471cc5" containerName="maas-api"
Jul 14 18:04:05.886981 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.886962    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"
Jul 14 18:04:05.889747 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.889718    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xxs8g\""
Jul 14 18:04:05.889911 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.889899    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 18:04:05.890507 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.890493    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8cb2afcb-serving-cert\""
Jul 14 18:04:05.890564 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.890537    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 18:04:05.900968 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.900945    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"]
Jul 14 18:04:05.928406 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.928373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:05.966531 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.966495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc45926e-baa5-4786-b798-c90f13e30091-openshift-service-ca\") pod \"maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf\" (UID: \"cc45926e-baa5-4786-b798-c90f13e30091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"
Jul 14 18:04:05.966697 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.966534    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc45926e-baa5-4786-b798-c90f13e30091-maas-api-tls\") pod \"maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf\" (UID: \"cc45926e-baa5-4786-b798-c90f13e30091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"
Jul 14 18:04:05.966697 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:05.966570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txdjw\" (UniqueName: \"kubernetes.io/projected/cc45926e-baa5-4786-b798-c90f13e30091-kube-api-access-txdjw\") pod \"maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf\" (UID: \"cc45926e-baa5-4786-b798-c90f13e30091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"
Jul 14 18:04:06.067243 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:06.067132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc45926e-baa5-4786-b798-c90f13e30091-openshift-service-ca\") pod \"maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf\" (UID: \"cc45926e-baa5-4786-b798-c90f13e30091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"
Jul 14 18:04:06.067243 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:06.067177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc45926e-baa5-4786-b798-c90f13e30091-maas-api-tls\") pod \"maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf\" (UID: \"cc45926e-baa5-4786-b798-c90f13e30091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"
Jul 14 18:04:06.067243 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:06.067232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txdjw\" (UniqueName: \"kubernetes.io/projected/cc45926e-baa5-4786-b798-c90f13e30091-kube-api-access-txdjw\") pod \"maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf\" (UID: \"cc45926e-baa5-4786-b798-c90f13e30091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"
Jul 14 18:04:06.067798 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:06.067775    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc45926e-baa5-4786-b798-c90f13e30091-openshift-service-ca\") pod \"maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf\" (UID: \"cc45926e-baa5-4786-b798-c90f13e30091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"
Jul 14 18:04:06.069772 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:06.069750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc45926e-baa5-4786-b798-c90f13e30091-maas-api-tls\") pod \"maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf\" (UID: \"cc45926e-baa5-4786-b798-c90f13e30091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"
Jul 14 18:04:06.095034 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:06.095006    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txdjw\" (UniqueName: \"kubernetes.io/projected/cc45926e-baa5-4786-b798-c90f13e30091-kube-api-access-txdjw\") pod \"maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf\" (UID: \"cc45926e-baa5-4786-b798-c90f13e30091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"
Jul 14 18:04:06.129921 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:06.129892    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:06.196944 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:06.196914    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"
Jul 14 18:04:06.348895 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:06.348871    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"]
Jul 14 18:04:06.350727 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:04:06.350704    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc45926e_baa5_4786_b798_c90f13e30091.slice/crio-f790d6d3e276c951f068f83d307274c767529f76ed25e6cfc9d05cfaa345c4e0 WatchSource:0}: Error finding container f790d6d3e276c951f068f83d307274c767529f76ed25e6cfc9d05cfaa345c4e0: Status 404 returned error can't find the container with id f790d6d3e276c951f068f83d307274c767529f76ed25e6cfc9d05cfaa345c4e0
Jul 14 18:04:06.351997 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:06.351980    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 18:04:06.771368 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:06.771332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:06.876469 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:06.876431    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:07.101965 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:07.101884    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf" event={"ID":"cc45926e-baa5-4786-b798-c90f13e30091","Type":"ContainerStarted","Data":"ecec537be73ad381ad327bbdce89209005b375c9b76d4856797a5c7841650870"}
Jul 14 18:04:07.101965 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:07.101934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf" event={"ID":"cc45926e-baa5-4786-b798-c90f13e30091","Type":"ContainerStarted","Data":"f790d6d3e276c951f068f83d307274c767529f76ed25e6cfc9d05cfaa345c4e0"}
Jul 14 18:04:07.102167 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:07.102052    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"
Jul 14 18:04:07.135657 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:07.135609    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf" podStartSLOduration=1.6262031810000002 podStartE2EDuration="2.135592603s" totalImagesPullingTime="509.389422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:04:07.13301202 +0000 UTC m=+1903.849405730" watchObservedRunningTime="2026-07-14 18:04:07.135592603 +0000 UTC m=+1903.851986330"
Jul 14 18:04:10.225301 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:10.225262    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:10.894419 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:10.894358    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:10.982508 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:10.982473    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:11.325903 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:11.325867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:12.664400 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:12.664360    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:04:13.112672 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:13.112648    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"
Jul 14 18:04:18.623480 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:18.623447    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:18.748826 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:18.748790    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:19.052459 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:19.052419    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd"]
Jul 14 18:04:19.056190 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:19.056175    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd"
Jul 14 18:04:19.059820 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:19.059800    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pd7w9\""
Jul 14 18:04:19.088083 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:19.088054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3f0ea4-3f24-460e-8478-daf0fe14b169-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd\" (UID: \"4a3f0ea4-3f24-460e-8478-daf0fe14b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd"
Jul 14 18:04:19.101388 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:19.101356    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd"]
Jul 14 18:04:19.189422 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:19.189389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3f0ea4-3f24-460e-8478-daf0fe14b169-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd\" (UID: \"4a3f0ea4-3f24-460e-8478-daf0fe14b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd"
Jul 14 18:04:19.190124 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:19.190103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3f0ea4-3f24-460e-8478-daf0fe14b169-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd\" (UID: \"4a3f0ea4-3f24-460e-8478-daf0fe14b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd"
Jul 14 18:04:19.367239 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:19.367113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd"
Jul 14 18:04:19.527500 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:04:19.527467    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3f0ea4_3f24_460e_8478_daf0fe14b169.slice/crio-e69849a7bac4badeb460921c040c43d3c67195c6e4ffb2cf2ce1fa4ece64b5f4 WatchSource:0}: Error finding container e69849a7bac4badeb460921c040c43d3c67195c6e4ffb2cf2ce1fa4ece64b5f4: Status 404 returned error can't find the container with id e69849a7bac4badeb460921c040c43d3c67195c6e4ffb2cf2ce1fa4ece64b5f4
Jul 14 18:04:19.529692 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:19.529670    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd"]
Jul 14 18:04:20.181388 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:20.181354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd" event={"ID":"4a3f0ea4-3f24-460e-8478-daf0fe14b169","Type":"ContainerStarted","Data":"62cc4faf48efa074362d4e1dc312daf9fbc541151017ee195e06d4fed7297ff5"}
Jul 14 18:04:20.181756 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:20.181402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd" event={"ID":"4a3f0ea4-3f24-460e-8478-daf0fe14b169","Type":"ContainerStarted","Data":"e69849a7bac4badeb460921c040c43d3c67195c6e4ffb2cf2ce1fa4ece64b5f4"}
Jul 14 18:04:20.227961 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:20.227907    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd" podStartSLOduration=1.22789071 podStartE2EDuration="1.22789071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:04:20.226515398 +0000 UTC m=+1916.942909090" watchObservedRunningTime="2026-07-14 18:04:20.22789071 +0000 UTC m=+1916.944284401"
Jul 14 18:04:21.188432 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:21.188392    2578 generic.go:356] "Generic (PLEG): container finished" podID="4a3f0ea4-3f24-460e-8478-daf0fe14b169" containerID="62cc4faf48efa074362d4e1dc312daf9fbc541151017ee195e06d4fed7297ff5" exitCode=0
Jul 14 18:04:21.188909 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:21.188481    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd" event={"ID":"4a3f0ea4-3f24-460e-8478-daf0fe14b169","Type":"ContainerDied","Data":"62cc4faf48efa074362d4e1dc312daf9fbc541151017ee195e06d4fed7297ff5"}
Jul 14 18:04:22.330420 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:22.330394    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd"
Jul 14 18:04:22.417407 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:22.417371    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3f0ea4-3f24-460e-8478-daf0fe14b169-maas-api-service-ca\") pod \"4a3f0ea4-3f24-460e-8478-daf0fe14b169\" (UID: \"4a3f0ea4-3f24-460e-8478-daf0fe14b169\") "
Jul 14 18:04:22.417741 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:22.417719    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3f0ea4-3f24-460e-8478-daf0fe14b169-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a3f0ea4-3f24-460e-8478-daf0fe14b169" (UID: "4a3f0ea4-3f24-460e-8478-daf0fe14b169"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:04:22.518315 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:22.518195    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3f0ea4-3f24-460e-8478-daf0fe14b169-maas-api-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:04:23.207353 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:23.207312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd" event={"ID":"4a3f0ea4-3f24-460e-8478-daf0fe14b169","Type":"ContainerDied","Data":"e69849a7bac4badeb460921c040c43d3c67195c6e4ffb2cf2ce1fa4ece64b5f4"}
Jul 14 18:04:23.207560 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:23.207353    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd"
Jul 14 18:04:23.207560 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:23.207389    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e69849a7bac4badeb460921c040c43d3c67195c6e4ffb2cf2ce1fa4ece64b5f4"
Jul 14 18:04:29.745895 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:29.745863    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"]
Jul 14 18:04:29.746308 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:29.746082    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf" podUID="cc45926e-baa5-4786-b798-c90f13e30091" containerName="maas-api" containerID="cri-o://ecec537be73ad381ad327bbdce89209005b375c9b76d4856797a5c7841650870" gracePeriod=30
Jul 14 18:04:29.795191 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:29.795154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:29.998980 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:29.998919    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"
Jul 14 18:04:30.082079 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.082042    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txdjw\" (UniqueName: \"kubernetes.io/projected/cc45926e-baa5-4786-b798-c90f13e30091-kube-api-access-txdjw\") pod \"cc45926e-baa5-4786-b798-c90f13e30091\" (UID: \"cc45926e-baa5-4786-b798-c90f13e30091\") "
Jul 14 18:04:30.082274 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.082103    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc45926e-baa5-4786-b798-c90f13e30091-maas-api-tls\") pod \"cc45926e-baa5-4786-b798-c90f13e30091\" (UID: \"cc45926e-baa5-4786-b798-c90f13e30091\") "
Jul 14 18:04:30.082274 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.082175    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc45926e-baa5-4786-b798-c90f13e30091-openshift-service-ca\") pod \"cc45926e-baa5-4786-b798-c90f13e30091\" (UID: \"cc45926e-baa5-4786-b798-c90f13e30091\") "
Jul 14 18:04:30.082513 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.082490    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc45926e-baa5-4786-b798-c90f13e30091-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc45926e-baa5-4786-b798-c90f13e30091" (UID: "cc45926e-baa5-4786-b798-c90f13e30091"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:04:30.084387 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.084358    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc45926e-baa5-4786-b798-c90f13e30091-kube-api-access-txdjw" (OuterVolumeSpecName: "kube-api-access-txdjw") pod "cc45926e-baa5-4786-b798-c90f13e30091" (UID: "cc45926e-baa5-4786-b798-c90f13e30091"). InnerVolumeSpecName "kube-api-access-txdjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:04:30.084387 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.084379    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc45926e-baa5-4786-b798-c90f13e30091-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc45926e-baa5-4786-b798-c90f13e30091" (UID: "cc45926e-baa5-4786-b798-c90f13e30091"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:04:30.183499 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.183468    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txdjw\" (UniqueName: \"kubernetes.io/projected/cc45926e-baa5-4786-b798-c90f13e30091-kube-api-access-txdjw\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:04:30.183499 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.183497    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc45926e-baa5-4786-b798-c90f13e30091-maas-api-tls\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:04:30.183499 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.183506    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc45926e-baa5-4786-b798-c90f13e30091-openshift-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:04:30.248741 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.248706    2578 generic.go:356] "Generic (PLEG): container finished" podID="cc45926e-baa5-4786-b798-c90f13e30091" containerID="ecec537be73ad381ad327bbdce89209005b375c9b76d4856797a5c7841650870" exitCode=0
Jul 14 18:04:30.248927 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.248751    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"
Jul 14 18:04:30.248927 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.248782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf" event={"ID":"cc45926e-baa5-4786-b798-c90f13e30091","Type":"ContainerDied","Data":"ecec537be73ad381ad327bbdce89209005b375c9b76d4856797a5c7841650870"}
Jul 14 18:04:30.248927 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.248837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf" event={"ID":"cc45926e-baa5-4786-b798-c90f13e30091","Type":"ContainerDied","Data":"f790d6d3e276c951f068f83d307274c767529f76ed25e6cfc9d05cfaa345c4e0"}
Jul 14 18:04:30.248927 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.248860    2578 scope.go:117] "RemoveContainer" containerID="ecec537be73ad381ad327bbdce89209005b375c9b76d4856797a5c7841650870"
Jul 14 18:04:30.259023 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.259002    2578 scope.go:117] "RemoveContainer" containerID="ecec537be73ad381ad327bbdce89209005b375c9b76d4856797a5c7841650870"
Jul 14 18:04:30.259340 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:04:30.259322    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecec537be73ad381ad327bbdce89209005b375c9b76d4856797a5c7841650870\": container with ID starting with ecec537be73ad381ad327bbdce89209005b375c9b76d4856797a5c7841650870 not found: ID does not exist" containerID="ecec537be73ad381ad327bbdce89209005b375c9b76d4856797a5c7841650870"
Jul 14 18:04:30.259417 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.259348    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecec537be73ad381ad327bbdce89209005b375c9b76d4856797a5c7841650870"} err="failed to get container status \"ecec537be73ad381ad327bbdce89209005b375c9b76d4856797a5c7841650870\": rpc error: code = NotFound desc = could not find container \"ecec537be73ad381ad327bbdce89209005b375c9b76d4856797a5c7841650870\": container with ID starting with ecec537be73ad381ad327bbdce89209005b375c9b76d4856797a5c7841650870 not found: ID does not exist"
Jul 14 18:04:30.275424 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.275395    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"]
Jul 14 18:04:30.282808 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:30.282779    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cb2afcb-6f6ddfb64f-s9wnf"]
Jul 14 18:04:31.783445 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:31.783404    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc45926e-baa5-4786-b798-c90f13e30091" path="/var/lib/kubelet/pods/cc45926e-baa5-4786-b798-c90f13e30091/volumes"
Jul 14 18:04:32.803237 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:32.803188    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:04:34.201073 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:34.201039    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:41.129220 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:41.129183    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:41.213974 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:41.213940    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:44.898498 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:44.898462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:45.135473 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:45.135440    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:45.259801 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:45.259763    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:45.358097 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:45.358068    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:45.494519 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:45.494488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:51.710341 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:51.710305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:51.833894 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:51.833845    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:51.902980 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:51.902942    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:52.062758 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:52.062673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:52.741724 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:52.741685    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:52.847161 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:52.847127    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:58.446475 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:58.446441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:04:58.523082 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:04:58.523049    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:05:07.111892 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:07.111853    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:05:07.177626 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:07.177591    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:05:08.868786 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:08.868750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:05:08.962048 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:08.962015    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:05:08.991549 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:08.991516    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"]
Jul 14 18:05:08.991904 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:08.991891    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a3f0ea4-3f24-460e-8478-daf0fe14b169" containerName="revoke-keys"
Jul 14 18:05:08.991949 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:08.991906    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a3f0ea4-3f24-460e-8478-daf0fe14b169" containerName="revoke-keys"
Jul 14 18:05:08.991949 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:08.991915    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc45926e-baa5-4786-b798-c90f13e30091" containerName="maas-api"
Jul 14 18:05:08.991949 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:08.991921    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc45926e-baa5-4786-b798-c90f13e30091" containerName="maas-api"
Jul 14 18:05:08.992043 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:08.991979    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc45926e-baa5-4786-b798-c90f13e30091" containerName="maas-api"
Jul 14 18:05:08.992043 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:08.991990    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a3f0ea4-3f24-460e-8478-daf0fe14b169" containerName="revoke-keys"
Jul 14 18:05:08.995062 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:08.995042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:08.998338 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:08.998318    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 18:05:08.998455 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:08.998400    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ef19c3e1-serving-cert\""
Jul 14 18:05:08.998455 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:08.998401    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 18:05:08.999242 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:08.999221    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xxs8g\""
Jul 14 18:05:09.013610 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:09.013581    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"]
Jul 14 18:05:09.023399 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:09.023366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-openshift-service-ca\") pod \"maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq\" (UID: \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:09.023523 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:09.023423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-maas-api-tls\") pod \"maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq\" (UID: \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:09.023523 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:09.023445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb2vw\" (UniqueName: \"kubernetes.io/projected/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-kube-api-access-lb2vw\") pod \"maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq\" (UID: \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:09.072960 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:09.072922    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:05:09.124755 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:09.124660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-maas-api-tls\") pod \"maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq\" (UID: \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:09.124755 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:09.124707    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb2vw\" (UniqueName: \"kubernetes.io/projected/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-kube-api-access-lb2vw\") pod \"maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq\" (UID: \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:09.124950 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:09.124769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-openshift-service-ca\") pod \"maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq\" (UID: \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:09.124950 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:05:09.124826    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-serving-cert: secret "maas-api-e2e-mt-ef19c3e1-serving-cert" not found
Jul 14 18:05:09.124950 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:05:09.124909    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-maas-api-tls podName:b69d4ee5-7d0f-4e44-bc62-378b83deccc5 nodeName:}" failed. No retries permitted until 2026-07-14 18:05:09.624886359 +0000 UTC m=+1966.341280045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-maas-api-tls") pod "maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq" (UID: "b69d4ee5-7d0f-4e44-bc62-378b83deccc5") : secret "maas-api-e2e-mt-ef19c3e1-serving-cert" not found
Jul 14 18:05:09.125383 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:09.125367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-openshift-service-ca\") pod \"maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq\" (UID: \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:09.140717 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:09.140688    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb2vw\" (UniqueName: \"kubernetes.io/projected/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-kube-api-access-lb2vw\") pod \"maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq\" (UID: \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:09.629132 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:09.629099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-maas-api-tls\") pod \"maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq\" (UID: \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:09.629394 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:05:09.629288    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-serving-cert: secret "maas-api-e2e-mt-ef19c3e1-serving-cert" not found
Jul 14 18:05:09.629394 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:05:09.629366    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-maas-api-tls podName:b69d4ee5-7d0f-4e44-bc62-378b83deccc5 nodeName:}" failed. No retries permitted until 2026-07-14 18:05:10.629344316 +0000 UTC m=+1967.345737986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-maas-api-tls") pod "maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq" (UID: "b69d4ee5-7d0f-4e44-bc62-378b83deccc5") : secret "maas-api-e2e-mt-ef19c3e1-serving-cert" not found
Jul 14 18:05:09.676015 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:09.675981    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:05:09.776257 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:09.776198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:05:10.638838 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:10.638800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-maas-api-tls\") pod \"maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq\" (UID: \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:10.641396 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:10.641373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-maas-api-tls\") pod \"maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq\" (UID: \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:10.805066 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:10.805021    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:10.949846 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:10.949773    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"]
Jul 14 18:05:10.951248 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:05:10.951218    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69d4ee5_7d0f_4e44_bc62_378b83deccc5.slice/crio-f5f8edbcdc5c946f3ee0fb7f98cd4c513367a19d026292800cb5c281ea085656 WatchSource:0}: Error finding container f5f8edbcdc5c946f3ee0fb7f98cd4c513367a19d026292800cb5c281ea085656: Status 404 returned error can't find the container with id f5f8edbcdc5c946f3ee0fb7f98cd4c513367a19d026292800cb5c281ea085656
Jul 14 18:05:11.497586 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:11.497547    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq" event={"ID":"b69d4ee5-7d0f-4e44-bc62-378b83deccc5","Type":"ContainerStarted","Data":"05f29e9d586a59ba58671d1bea1c98f341e6e92bdd7e8c519544dc00af6dbec6"}
Jul 14 18:05:11.497763 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:11.497647    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq" event={"ID":"b69d4ee5-7d0f-4e44-bc62-378b83deccc5","Type":"ContainerStarted","Data":"f5f8edbcdc5c946f3ee0fb7f98cd4c513367a19d026292800cb5c281ea085656"}
Jul 14 18:05:11.497838 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:11.497779    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:11.530802 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:11.530751    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq" podStartSLOduration=3.081793135 podStartE2EDuration="3.530735841s" totalImagesPullingTime="448.942706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:05:11.527951209 +0000 UTC m=+1968.244344929" watchObservedRunningTime="2026-07-14 18:05:11.530735841 +0000 UTC m=+1968.247129533"
Jul 14 18:05:13.365591 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:13.365559    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:05:15.399965 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:15.399887    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:05:17.508619 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:17.508592    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:19.465258 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.465221    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"]
Jul 14 18:05:19.469177 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.469153    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.471860 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.471839    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ef19c3e1\"/\"default-dockercfg-ntpcj\""
Jul 14 18:05:19.472584 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.472565    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ef19c3e1\"/\"kube-root-ca.crt\""
Jul 14 18:05:19.473091 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.473070    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ef19c3e1\"/\"openshift-service-ca.crt\""
Jul 14 18:05:19.473505 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.473489    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ef19c3e1\"/\"e2e-lifecycle-model-ef19c3e1-kserve-self-signed-certs\""
Jul 14 18:05:19.485073 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.485048    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"]
Jul 14 18:05:19.522019 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.521986    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:05:19.525143 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.525113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9zgt\" (UniqueName: \"kubernetes.io/projected/dad556fb-ae31-4369-aa18-2cea3aef458d-kube-api-access-d9zgt\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.525297 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.525152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad556fb-ae31-4369-aa18-2cea3aef458d-tls-certs\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.525297 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.525287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-tmp-dir\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.525390 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.525334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-home\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.525390 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.525353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-model-cache\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.525498 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.525412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-dshm\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.625851 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.625814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-tmp-dir\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.626034 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.625871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-home\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.626034 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.625970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-model-cache\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.626034 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.626013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-dshm\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.626251 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.626082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9zgt\" (UniqueName: \"kubernetes.io/projected/dad556fb-ae31-4369-aa18-2cea3aef458d-kube-api-access-d9zgt\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.626251 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.626110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad556fb-ae31-4369-aa18-2cea3aef458d-tls-certs\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.626251 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.626173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-tmp-dir\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.626397 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.626250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-home\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.626397 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.626323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-model-cache\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.628658 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.628624    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-dshm\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.628794 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.628778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad556fb-ae31-4369-aa18-2cea3aef458d-tls-certs\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.636176 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.636148    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9zgt\" (UniqueName: \"kubernetes.io/projected/dad556fb-ae31-4369-aa18-2cea3aef458d-kube-api-access-d9zgt\") pod \"e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") " pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.778116 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.778044    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:19.922647 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:19.922603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"]
Jul 14 18:05:19.926638 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:05:19.926609    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddad556fb_ae31_4369_aa18_2cea3aef458d.slice/crio-ea861febd5a8c4424a187db83167b92ceb490d1b9abd109e0d0853247f2e7b07 WatchSource:0}: Error finding container ea861febd5a8c4424a187db83167b92ceb490d1b9abd109e0d0853247f2e7b07: Status 404 returned error can't find the container with id ea861febd5a8c4424a187db83167b92ceb490d1b9abd109e0d0853247f2e7b07
Jul 14 18:05:20.552018 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:20.551982    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x" event={"ID":"dad556fb-ae31-4369-aa18-2cea3aef458d","Type":"ContainerStarted","Data":"9133b28c12254d421cdeae0a49441f20c04fde5cb7e3eb229b9c088ffdd0beb8"}
Jul 14 18:05:20.552018 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:20.552028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x" event={"ID":"dad556fb-ae31-4369-aa18-2cea3aef458d","Type":"ContainerStarted","Data":"ea861febd5a8c4424a187db83167b92ceb490d1b9abd109e0d0853247f2e7b07"}
Jul 14 18:05:20.579822 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:20.579754    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x" podStartSLOduration=1.579733059 podStartE2EDuration="1.579733059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:05:20.575448399 +0000 UTC m=+1977.291842111" watchObservedRunningTime="2026-07-14 18:05:20.579733059 +0000 UTC m=+1977.296126752"
Jul 14 18:05:24.159568 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:24.159531    2578 scope.go:117] "RemoveContainer" containerID="74f7f8fd48ef89a370924727d12ae257cff74400e909ee4ee998b592e62c19b7"
Jul 14 18:05:24.432462 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:24.432430    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:05:26.571339 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:26.571301    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:05:26.672615 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:26.672581    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:05:29.784047 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:29.784021    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:29.784444 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:29.784067    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:29.792550 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:29.792527    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:30.624079 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:30.624047    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:05:31.816110 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:31.816084    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:05:33.573555 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:33.573518    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:05:38.231091 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:38.231053    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:05:41.451484 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:41.451441    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2"]
Jul 14 18:05:41.455222 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:41.455182    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2"
Jul 14 18:05:41.461885 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:41.461863    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pd7w9\""
Jul 14 18:05:41.483480 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:41.483456    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2"]
Jul 14 18:05:41.626888 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:41.626852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6780df0-ec66-463c-a21c-5ad5360b5b68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2\" (UID: \"e6780df0-ec66-463c-a21c-5ad5360b5b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2"
Jul 14 18:05:41.728358 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:41.728270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6780df0-ec66-463c-a21c-5ad5360b5b68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2\" (UID: \"e6780df0-ec66-463c-a21c-5ad5360b5b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2"
Jul 14 18:05:41.728921 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:41.728900    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6780df0-ec66-463c-a21c-5ad5360b5b68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2\" (UID: \"e6780df0-ec66-463c-a21c-5ad5360b5b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2"
Jul 14 18:05:41.764807 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:41.764770    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2"
Jul 14 18:05:41.911297 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:41.911272    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2"]
Jul 14 18:05:41.912843 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:05:41.912816    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6780df0_ec66_463c_a21c_5ad5360b5b68.slice/crio-17fba723338f5ea23b185256aa8f19ded1d5f9b2719bcbc393b7f76ec32712ef WatchSource:0}: Error finding container 17fba723338f5ea23b185256aa8f19ded1d5f9b2719bcbc393b7f76ec32712ef: Status 404 returned error can't find the container with id 17fba723338f5ea23b185256aa8f19ded1d5f9b2719bcbc393b7f76ec32712ef
Jul 14 18:05:42.686835 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:42.686796    2578 generic.go:356] "Generic (PLEG): container finished" podID="e6780df0-ec66-463c-a21c-5ad5360b5b68" containerID="ad3cf8b5fcad0abc589fc34ffaac8eaf30c9309e235d86784baac77569f8aeb0" exitCode=0
Jul 14 18:05:42.687200 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:42.686896    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2" event={"ID":"e6780df0-ec66-463c-a21c-5ad5360b5b68","Type":"ContainerDied","Data":"ad3cf8b5fcad0abc589fc34ffaac8eaf30c9309e235d86784baac77569f8aeb0"}
Jul 14 18:05:42.687200 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:42.686941    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2" event={"ID":"e6780df0-ec66-463c-a21c-5ad5360b5b68","Type":"ContainerStarted","Data":"17fba723338f5ea23b185256aa8f19ded1d5f9b2719bcbc393b7f76ec32712ef"}
Jul 14 18:05:43.838504 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:43.838476    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2"
Jul 14 18:05:43.951416 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:43.951320    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6780df0-ec66-463c-a21c-5ad5360b5b68-maas-api-service-ca\") pod \"e6780df0-ec66-463c-a21c-5ad5360b5b68\" (UID: \"e6780df0-ec66-463c-a21c-5ad5360b5b68\") "
Jul 14 18:05:43.951719 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:43.951693    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6780df0-ec66-463c-a21c-5ad5360b5b68-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e6780df0-ec66-463c-a21c-5ad5360b5b68" (UID: "e6780df0-ec66-463c-a21c-5ad5360b5b68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:05:44.052719 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:44.052683    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6780df0-ec66-463c-a21c-5ad5360b5b68-maas-api-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:05:44.701085 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:44.701059    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2"
Jul 14 18:05:44.701274 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:44.701098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2" event={"ID":"e6780df0-ec66-463c-a21c-5ad5360b5b68","Type":"ContainerDied","Data":"17fba723338f5ea23b185256aa8f19ded1d5f9b2719bcbc393b7f76ec32712ef"}
Jul 14 18:05:44.701274 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:44.701142    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17fba723338f5ea23b185256aa8f19ded1d5f9b2719bcbc393b7f76ec32712ef"
Jul 14 18:05:52.286156 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.286122    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"]
Jul 14 18:05:52.286574 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.286363    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq" podUID="b69d4ee5-7d0f-4e44-bc62-378b83deccc5" containerName="maas-api" containerID="cri-o://05f29e9d586a59ba58671d1bea1c98f341e6e92bdd7e8c519544dc00af6dbec6" gracePeriod=30
Jul 14 18:05:52.309222 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.309167    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:05:52.548042 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.547983    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:52.625236 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.625177    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-openshift-service-ca\") pod \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\" (UID: \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\") "
Jul 14 18:05:52.625406 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.625276    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-maas-api-tls\") pod \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\" (UID: \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\") "
Jul 14 18:05:52.625406 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.625295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb2vw\" (UniqueName: \"kubernetes.io/projected/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-kube-api-access-lb2vw\") pod \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\" (UID: \"b69d4ee5-7d0f-4e44-bc62-378b83deccc5\") "
Jul 14 18:05:52.625606 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.625579    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b69d4ee5-7d0f-4e44-bc62-378b83deccc5" (UID: "b69d4ee5-7d0f-4e44-bc62-378b83deccc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:05:52.627508 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.627485    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-kube-api-access-lb2vw" (OuterVolumeSpecName: "kube-api-access-lb2vw") pod "b69d4ee5-7d0f-4e44-bc62-378b83deccc5" (UID: "b69d4ee5-7d0f-4e44-bc62-378b83deccc5"). InnerVolumeSpecName "kube-api-access-lb2vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:05:52.627596 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.627500    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b69d4ee5-7d0f-4e44-bc62-378b83deccc5" (UID: "b69d4ee5-7d0f-4e44-bc62-378b83deccc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:05:52.726770 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.726736    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-openshift-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:05:52.726770 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.726769    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-maas-api-tls\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:05:52.726770 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.726778    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb2vw\" (UniqueName: \"kubernetes.io/projected/b69d4ee5-7d0f-4e44-bc62-378b83deccc5-kube-api-access-lb2vw\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:05:52.748257 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.748194    2578 generic.go:356] "Generic (PLEG): container finished" podID="b69d4ee5-7d0f-4e44-bc62-378b83deccc5" containerID="05f29e9d586a59ba58671d1bea1c98f341e6e92bdd7e8c519544dc00af6dbec6" exitCode=0
Jul 14 18:05:52.748442 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.748270    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"
Jul 14 18:05:52.748442 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.748311    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq" event={"ID":"b69d4ee5-7d0f-4e44-bc62-378b83deccc5","Type":"ContainerDied","Data":"05f29e9d586a59ba58671d1bea1c98f341e6e92bdd7e8c519544dc00af6dbec6"}
Jul 14 18:05:52.748442 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.748370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq" event={"ID":"b69d4ee5-7d0f-4e44-bc62-378b83deccc5","Type":"ContainerDied","Data":"f5f8edbcdc5c946f3ee0fb7f98cd4c513367a19d026292800cb5c281ea085656"}
Jul 14 18:05:52.748442 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.748403    2578 scope.go:117] "RemoveContainer" containerID="05f29e9d586a59ba58671d1bea1c98f341e6e92bdd7e8c519544dc00af6dbec6"
Jul 14 18:05:52.758477 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.758461    2578 scope.go:117] "RemoveContainer" containerID="05f29e9d586a59ba58671d1bea1c98f341e6e92bdd7e8c519544dc00af6dbec6"
Jul 14 18:05:52.758736 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:05:52.758717    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f29e9d586a59ba58671d1bea1c98f341e6e92bdd7e8c519544dc00af6dbec6\": container with ID starting with 05f29e9d586a59ba58671d1bea1c98f341e6e92bdd7e8c519544dc00af6dbec6 not found: ID does not exist" containerID="05f29e9d586a59ba58671d1bea1c98f341e6e92bdd7e8c519544dc00af6dbec6"
Jul 14 18:05:52.758783 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.758746    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f29e9d586a59ba58671d1bea1c98f341e6e92bdd7e8c519544dc00af6dbec6"} err="failed to get container status \"05f29e9d586a59ba58671d1bea1c98f341e6e92bdd7e8c519544dc00af6dbec6\": rpc error: code = NotFound desc = could not find container \"05f29e9d586a59ba58671d1bea1c98f341e6e92bdd7e8c519544dc00af6dbec6\": container with ID starting with 05f29e9d586a59ba58671d1bea1c98f341e6e92bdd7e8c519544dc00af6dbec6 not found: ID does not exist"
Jul 14 18:05:52.773311 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.773284    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"]
Jul 14 18:05:52.776959 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:52.776936    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq"]
Jul 14 18:05:53.782738 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:53.782707    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b69d4ee5-7d0f-4e44-bc62-378b83deccc5" path="/var/lib/kubelet/pods/b69d4ee5-7d0f-4e44-bc62-378b83deccc5/volumes"
Jul 14 18:05:55.505582 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:55.505521    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef19c3e1-f768cfb47-pwglq" podUID="b69d4ee5-7d0f-4e44-bc62-378b83deccc5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.63:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 14 18:05:55.629666 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:55.629630    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:05:56.684963 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:05:56.684903    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:06:01.626664 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:01.626626    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"]
Jul 14 18:06:01.749160 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:01.749121    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:06:16.647230 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:16.647142    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x" podUID="dad556fb-ae31-4369-aa18-2cea3aef458d" containerName="main" containerID="cri-o://9133b28c12254d421cdeae0a49441f20c04fde5cb7e3eb229b9c088ffdd0beb8" gracePeriod=45
Jul 14 18:06:16.889339 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:16.889310    2578 generic.go:356] "Generic (PLEG): container finished" podID="dad556fb-ae31-4369-aa18-2cea3aef458d" containerID="9133b28c12254d421cdeae0a49441f20c04fde5cb7e3eb229b9c088ffdd0beb8" exitCode=0
Jul 14 18:06:16.889476 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:16.889395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x" event={"ID":"dad556fb-ae31-4369-aa18-2cea3aef458d","Type":"ContainerDied","Data":"9133b28c12254d421cdeae0a49441f20c04fde5cb7e3eb229b9c088ffdd0beb8"}
Jul 14 18:06:16.889476 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:16.889454    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x" event={"ID":"dad556fb-ae31-4369-aa18-2cea3aef458d","Type":"ContainerDied","Data":"ea861febd5a8c4424a187db83167b92ceb490d1b9abd109e0d0853247f2e7b07"}
Jul 14 18:06:16.889551 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:16.889474    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea861febd5a8c4424a187db83167b92ceb490d1b9abd109e0d0853247f2e7b07"
Jul 14 18:06:16.898809 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:16.898755    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:06:17.041185 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.041153    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad556fb-ae31-4369-aa18-2cea3aef458d-tls-certs\") pod \"dad556fb-ae31-4369-aa18-2cea3aef458d\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") "
Jul 14 18:06:17.041404 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.041196    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9zgt\" (UniqueName: \"kubernetes.io/projected/dad556fb-ae31-4369-aa18-2cea3aef458d-kube-api-access-d9zgt\") pod \"dad556fb-ae31-4369-aa18-2cea3aef458d\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") "
Jul 14 18:06:17.041404 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.041249    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-tmp-dir\") pod \"dad556fb-ae31-4369-aa18-2cea3aef458d\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") "
Jul 14 18:06:17.041404 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.041298    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-model-cache\") pod \"dad556fb-ae31-4369-aa18-2cea3aef458d\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") "
Jul 14 18:06:17.041404 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.041332    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-dshm\") pod \"dad556fb-ae31-4369-aa18-2cea3aef458d\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") "
Jul 14 18:06:17.041404 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.041353    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-home\") pod \"dad556fb-ae31-4369-aa18-2cea3aef458d\" (UID: \"dad556fb-ae31-4369-aa18-2cea3aef458d\") "
Jul 14 18:06:17.041906 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.041512    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dad556fb-ae31-4369-aa18-2cea3aef458d" (UID: "dad556fb-ae31-4369-aa18-2cea3aef458d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:06:17.041906 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.041665    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-home" (OuterVolumeSpecName: "home") pod "dad556fb-ae31-4369-aa18-2cea3aef458d" (UID: "dad556fb-ae31-4369-aa18-2cea3aef458d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:06:17.041906 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.041681    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-model-cache" (OuterVolumeSpecName: "model-cache") pod "dad556fb-ae31-4369-aa18-2cea3aef458d" (UID: "dad556fb-ae31-4369-aa18-2cea3aef458d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:06:17.043532 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.043507    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad556fb-ae31-4369-aa18-2cea3aef458d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dad556fb-ae31-4369-aa18-2cea3aef458d" (UID: "dad556fb-ae31-4369-aa18-2cea3aef458d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:06:17.043639 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.043596    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-dshm" (OuterVolumeSpecName: "dshm") pod "dad556fb-ae31-4369-aa18-2cea3aef458d" (UID: "dad556fb-ae31-4369-aa18-2cea3aef458d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:06:17.043703 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.043634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad556fb-ae31-4369-aa18-2cea3aef458d-kube-api-access-d9zgt" (OuterVolumeSpecName: "kube-api-access-d9zgt") pod "dad556fb-ae31-4369-aa18-2cea3aef458d" (UID: "dad556fb-ae31-4369-aa18-2cea3aef458d"). InnerVolumeSpecName "kube-api-access-d9zgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:06:17.142637 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.142590    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad556fb-ae31-4369-aa18-2cea3aef458d-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:06:17.142637 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.142633    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9zgt\" (UniqueName: \"kubernetes.io/projected/dad556fb-ae31-4369-aa18-2cea3aef458d-kube-api-access-d9zgt\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:06:17.142637 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.142645    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:06:17.142637 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.142655    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:06:17.142913 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.142663    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:06:17.142913 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.142670    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad556fb-ae31-4369-aa18-2cea3aef458d-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:06:17.895707 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.895665    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"
Jul 14 18:06:17.919760 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.919723    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"]
Jul 14 18:06:17.924186 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:17.924157    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ef19c3e1/e2e-lifecycle-model-ef19c3e1-kserve-8565d785c9-wk44x"]
Jul 14 18:06:19.783319 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:19.783287    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dad556fb-ae31-4369-aa18-2cea3aef458d" path="/var/lib/kubelet/pods/dad556fb-ae31-4369-aa18-2cea3aef458d/volumes"
Jul 14 18:06:24.186971 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:24.186935    2578 scope.go:117] "RemoveContainer" containerID="b16292e59d0862e8b15cc839f8b01afe2d9834b21fffc8ea8dac4fc43f633d8e"
Jul 14 18:06:24.195981 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:24.195960    2578 scope.go:117] "RemoveContainer" containerID="3edca1ddebd4069a4e645cc1deb7ff0ea9e0bd4a43f6ce4f8dcedbb84a7d1825"
Jul 14 18:06:37.874077 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:37.874044    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:06:41.558703 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:41.558670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:06:49.211424 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:49.211338    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:06:49.310358 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:49.310323    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:06:59.070309 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:59.070255    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:06:59.287975 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:06:59.287938    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:00.348318 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:00.348267    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:00.441613 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:00.441585    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:00.549762 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:00.549727    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:01.422534 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:01.422504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:01.565565 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:01.565528    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:02.727635 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:02.727602    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:07:04.198709 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:04.198674    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:09.903031 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:09.902991    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:10.002405 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:10.002372    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:10.633827 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:10.633789    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:14.808819 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:14.808779    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:07:18.332406 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:18.332377    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:18.435064 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:18.435032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:18.541540 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:18.541513    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:24.224136 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:24.224099    2578 scope.go:117] "RemoveContainer" containerID="4a67520b7796daec3a31578901327e88ad15d53703a754178d13ad62a97dfb13"
Jul 14 18:07:24.337225 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:24.337098    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 18:07:24.343751 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:24.340396    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 18:07:29.108218 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:29.108181    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:29.477448 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:29.477416    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:50.434810 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:50.434780    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:57.377163 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:57.377128    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:57.515382 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:57.515349    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:57.736363 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:57.736332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:58.105684 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:58.105572    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:07:58.254741 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:07:58.254697    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:01.739902 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.739853    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:01.835540 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.835510    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"]
Jul 14 18:08:01.835901 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.835887    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6780df0-ec66-463c-a21c-5ad5360b5b68" containerName="revoke-keys"
Jul 14 18:08:01.835945 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.835903    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6780df0-ec66-463c-a21c-5ad5360b5b68" containerName="revoke-keys"
Jul 14 18:08:01.835945 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.835917    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad556fb-ae31-4369-aa18-2cea3aef458d" containerName="main"
Jul 14 18:08:01.835945 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.835923    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad556fb-ae31-4369-aa18-2cea3aef458d" containerName="main"
Jul 14 18:08:01.835945 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.835943    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b69d4ee5-7d0f-4e44-bc62-378b83deccc5" containerName="maas-api"
Jul 14 18:08:01.836071 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.835949    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b69d4ee5-7d0f-4e44-bc62-378b83deccc5" containerName="maas-api"
Jul 14 18:08:01.836071 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.836006    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6780df0-ec66-463c-a21c-5ad5360b5b68" containerName="revoke-keys"
Jul 14 18:08:01.836071 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.836016    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b69d4ee5-7d0f-4e44-bc62-378b83deccc5" containerName="maas-api"
Jul 14 18:08:01.836071 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.836023    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dad556fb-ae31-4369-aa18-2cea3aef458d" containerName="main"
Jul 14 18:08:01.839267 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.839247    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:08:01.842374 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.842353    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xxs8g\""
Jul 14 18:08:01.842495 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.842403    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ae8cb8-serving-cert\""
Jul 14 18:08:01.843028 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.843015    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 18:08:01.845325 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.845302    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 18:08:01.854617 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.854594    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"]
Jul 14 18:08:01.884373 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.884334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:01.958010 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.957973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402f5976-86cc-402a-b0ad-91176a60e45b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9\" (UID: \"402f5976-86cc-402a-b0ad-91176a60e45b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:08:01.958236 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.958043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r46l\" (UniqueName: \"kubernetes.io/projected/402f5976-86cc-402a-b0ad-91176a60e45b-kube-api-access-5r46l\") pod \"maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9\" (UID: \"402f5976-86cc-402a-b0ad-91176a60e45b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:08:01.958236 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:01.958101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402f5976-86cc-402a-b0ad-91176a60e45b-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9\" (UID: \"402f5976-86cc-402a-b0ad-91176a60e45b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:08:02.058969 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:02.058870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402f5976-86cc-402a-b0ad-91176a60e45b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9\" (UID: \"402f5976-86cc-402a-b0ad-91176a60e45b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:08:02.058969 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:02.058951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r46l\" (UniqueName: \"kubernetes.io/projected/402f5976-86cc-402a-b0ad-91176a60e45b-kube-api-access-5r46l\") pod \"maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9\" (UID: \"402f5976-86cc-402a-b0ad-91176a60e45b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:08:02.059304 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:02.059014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402f5976-86cc-402a-b0ad-91176a60e45b-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9\" (UID: \"402f5976-86cc-402a-b0ad-91176a60e45b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:08:02.059304 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:08:02.059120    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-serving-cert: secret "maas-api-e2e-inf-a-ae8cb8-serving-cert" not found
Jul 14 18:08:02.059304 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:08:02.059190    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/402f5976-86cc-402a-b0ad-91176a60e45b-maas-api-tls podName:402f5976-86cc-402a-b0ad-91176a60e45b nodeName:}" failed. No retries permitted until 2026-07-14 18:08:02.559168321 +0000 UTC m=+2139.275561992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/402f5976-86cc-402a-b0ad-91176a60e45b-maas-api-tls") pod "maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9" (UID: "402f5976-86cc-402a-b0ad-91176a60e45b") : secret "maas-api-e2e-inf-a-ae8cb8-serving-cert" not found
Jul 14 18:08:02.059616 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:02.059594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402f5976-86cc-402a-b0ad-91176a60e45b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9\" (UID: \"402f5976-86cc-402a-b0ad-91176a60e45b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:08:02.076368 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:02.076333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r46l\" (UniqueName: \"kubernetes.io/projected/402f5976-86cc-402a-b0ad-91176a60e45b-kube-api-access-5r46l\") pod \"maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9\" (UID: \"402f5976-86cc-402a-b0ad-91176a60e45b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:08:02.093396 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:02.093362    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:02.564438 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:02.564406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402f5976-86cc-402a-b0ad-91176a60e45b-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9\" (UID: \"402f5976-86cc-402a-b0ad-91176a60e45b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:08:02.567237 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:02.567191    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402f5976-86cc-402a-b0ad-91176a60e45b-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9\" (UID: \"402f5976-86cc-402a-b0ad-91176a60e45b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:08:02.591632 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:02.591589    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:02.701138 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:02.701103    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:02.749446 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:02.749416    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:08:02.804295 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:02.804154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:02.906369 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:02.906339    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"]
Jul 14 18:08:02.907877 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:08:02.907852    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402f5976_86cc_402a_b0ad_91176a60e45b.slice/crio-d8c1dc530e6b899cb60a3b452cf7132b5739404c0913af04ba0ed2e07f6c0142 WatchSource:0}: Error finding container d8c1dc530e6b899cb60a3b452cf7132b5739404c0913af04ba0ed2e07f6c0142: Status 404 returned error can't find the container with id d8c1dc530e6b899cb60a3b452cf7132b5739404c0913af04ba0ed2e07f6c0142
Jul 14 18:08:03.505646 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:03.505609    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9" event={"ID":"402f5976-86cc-402a-b0ad-91176a60e45b","Type":"ContainerStarted","Data":"d8c1dc530e6b899cb60a3b452cf7132b5739404c0913af04ba0ed2e07f6c0142"}
Jul 14 18:08:05.518409 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:05.518321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9" event={"ID":"402f5976-86cc-402a-b0ad-91176a60e45b","Type":"ContainerStarted","Data":"0acea30460473503edbc87b85f3ca21f6c49611b3b8a387c6eb3f1efa6b0e732"}
Jul 14 18:08:05.518854 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:05.518417    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:08:05.542390 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:05.542344    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9" podStartSLOduration=2.271403938 podStartE2EDuration="4.54232763s" totalImagesPullingTime="2.270923692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:08:05.54000389 +0000 UTC m=+2142.256397701" watchObservedRunningTime="2026-07-14 18:08:05.54232763 +0000 UTC m=+2142.258721322"
Jul 14 18:08:06.334188 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:06.334156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:06.470132 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:06.470099    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:07.236602 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:07.236566    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:07.346266 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:07.346199    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:08.005692 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:08.005657    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:08:09.683413 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:09.683358    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:09.787519 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:09.787488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:09.826910 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:09.826873    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"]
Jul 14 18:08:09.830512 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:09.830491    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:08:09.833022 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:09.832996    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9c28f2-serving-cert\""
Jul 14 18:08:09.849636 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:09.849611    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"]
Jul 14 18:08:09.884357 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:09.884328    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:09.933320 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:09.933287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d23e6a4a-47cc-423e-ba42-5330dec64e88-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t\" (UID: \"d23e6a4a-47cc-423e-ba42-5330dec64e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:08:09.933523 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:09.933326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d23e6a4a-47cc-423e-ba42-5330dec64e88-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t\" (UID: \"d23e6a4a-47cc-423e-ba42-5330dec64e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:08:09.933523 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:09.933406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n5xd\" (UniqueName: \"kubernetes.io/projected/d23e6a4a-47cc-423e-ba42-5330dec64e88-kube-api-access-5n5xd\") pod \"maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t\" (UID: \"d23e6a4a-47cc-423e-ba42-5330dec64e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:08:10.034635 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:10.034601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d23e6a4a-47cc-423e-ba42-5330dec64e88-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t\" (UID: \"d23e6a4a-47cc-423e-ba42-5330dec64e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:08:10.034819 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:10.034642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d23e6a4a-47cc-423e-ba42-5330dec64e88-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t\" (UID: \"d23e6a4a-47cc-423e-ba42-5330dec64e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:08:10.034819 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:10.034686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n5xd\" (UniqueName: \"kubernetes.io/projected/d23e6a4a-47cc-423e-ba42-5330dec64e88-kube-api-access-5n5xd\") pod \"maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t\" (UID: \"d23e6a4a-47cc-423e-ba42-5330dec64e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:08:10.034819 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:08:10.034758    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-serving-cert: secret "maas-api-e2e-inf-b-9c28f2-serving-cert" not found
Jul 14 18:08:10.034933 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:08:10.034834    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23e6a4a-47cc-423e-ba42-5330dec64e88-maas-api-tls podName:d23e6a4a-47cc-423e-ba42-5330dec64e88 nodeName:}" failed. No retries permitted until 2026-07-14 18:08:10.534815229 +0000 UTC m=+2147.251208899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d23e6a4a-47cc-423e-ba42-5330dec64e88-maas-api-tls") pod "maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t" (UID: "d23e6a4a-47cc-423e-ba42-5330dec64e88") : secret "maas-api-e2e-inf-b-9c28f2-serving-cert" not found
Jul 14 18:08:10.035351 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:10.035333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d23e6a4a-47cc-423e-ba42-5330dec64e88-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t\" (UID: \"d23e6a4a-47cc-423e-ba42-5330dec64e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:08:10.049196 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:10.049162    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n5xd\" (UniqueName: \"kubernetes.io/projected/d23e6a4a-47cc-423e-ba42-5330dec64e88-kube-api-access-5n5xd\") pod \"maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t\" (UID: \"d23e6a4a-47cc-423e-ba42-5330dec64e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:08:10.466531 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:10.466503    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:10.539908 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:10.539868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d23e6a4a-47cc-423e-ba42-5330dec64e88-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t\" (UID: \"d23e6a4a-47cc-423e-ba42-5330dec64e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:08:10.542592 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:10.542566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d23e6a4a-47cc-423e-ba42-5330dec64e88-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t\" (UID: \"d23e6a4a-47cc-423e-ba42-5330dec64e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:08:10.567505 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:10.567476    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:10.663550 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:10.663517    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:10.741432 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:10.741344    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:08:11.097603 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:11.097576    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"]
Jul 14 18:08:11.098028 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:08:11.097999    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23e6a4a_47cc_423e_ba42_5330dec64e88.slice/crio-3230944d4836affc9e389f86121e4141915dd798fdee28b7ad34b029653034df WatchSource:0}: Error finding container 3230944d4836affc9e389f86121e4141915dd798fdee28b7ad34b029653034df: Status 404 returned error can't find the container with id 3230944d4836affc9e389f86121e4141915dd798fdee28b7ad34b029653034df
Jul 14 18:08:11.533643 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:11.533621    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:08:11.560078 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:11.560048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t" event={"ID":"d23e6a4a-47cc-423e-ba42-5330dec64e88","Type":"ContainerStarted","Data":"733727e25516f4c478958e4532151e5cc37244b8bfd24c451d480eb5edc67c79"}
Jul 14 18:08:11.560227 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:11.560096    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t" event={"ID":"d23e6a4a-47cc-423e-ba42-5330dec64e88","Type":"ContainerStarted","Data":"3230944d4836affc9e389f86121e4141915dd798fdee28b7ad34b029653034df"}
Jul 14 18:08:11.560227 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:11.560185    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:08:11.587292 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:11.587245    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t" podStartSLOduration=2.218083273 podStartE2EDuration="2.587198231s" totalImagesPullingTime="369.114958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:08:11.585887536 +0000 UTC m=+2148.302281229" watchObservedRunningTime="2026-07-14 18:08:11.587198231 +0000 UTC m=+2148.303591924"
Jul 14 18:08:14.145154 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:14.145069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:17.571231 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:17.571177    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:08:19.445609 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:19.445567    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:08:19.917942 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:19.917837    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:23.791681 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:23.791650    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:23.910594 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:23.910561    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:28.447791 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.447753    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"]
Jul 14 18:08:28.452383 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.452325    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.456631 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.456596    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ae8cb8\"/\"kube-root-ca.crt\""
Jul 14 18:08:28.457452 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.457430    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ae8cb8\"/\"openshift-service-ca.crt\""
Jul 14 18:08:28.457575 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.457475    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ae8cb8\"/\"default-dockercfg-6269c\""
Jul 14 18:08:28.457575 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.457520    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ae8cb8\"/\"test-model-ae8cb8-kserve-self-signed-certs\""
Jul 14 18:08:28.464000 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.463980    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"]
Jul 14 18:08:28.499259 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.499215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-home\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.499259 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.499262    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-dshm\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.499486 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.499348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-model-cache\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.499486 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.499382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb706b6c-f43d-4b35-a511-1d9a9221b192-tls-certs\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.499486 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.499418    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-tmp-dir\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.499486 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.499443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2dl\" (UniqueName: \"kubernetes.io/projected/bb706b6c-f43d-4b35-a511-1d9a9221b192-kube-api-access-sh2dl\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.512944 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.512907    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:28.600141 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.600106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-home\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.600141 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.600144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-dshm\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.600416 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.600180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-model-cache\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.600416 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.600200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb706b6c-f43d-4b35-a511-1d9a9221b192-tls-certs\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.600416 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.600343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-tmp-dir\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.600416 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.600387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2dl\" (UniqueName: \"kubernetes.io/projected/bb706b6c-f43d-4b35-a511-1d9a9221b192-kube-api-access-sh2dl\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.600633 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.600536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-home\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.600691 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.600653    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-model-cache\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.600731 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.600694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-tmp-dir\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.602724 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.602703    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-dshm\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.602857 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.602795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb706b6c-f43d-4b35-a511-1d9a9221b192-tls-certs\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.611021 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.611000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2dl\" (UniqueName: \"kubernetes.io/projected/bb706b6c-f43d-4b35-a511-1d9a9221b192-kube-api-access-sh2dl\") pod \"test-model-ae8cb8-kserve-98c4f94dc-c7tnt\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") " pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.764281 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.764168    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:28.910891 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:28.910861    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"]
Jul 14 18:08:28.912006 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:08:28.911978    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb706b6c_f43d_4b35_a511_1d9a9221b192.slice/crio-069b3af5291ef77dbd2dad03b1fc781133a5432bf0ced48459a1c3d0fe778874 WatchSource:0}: Error finding container 069b3af5291ef77dbd2dad03b1fc781133a5432bf0ced48459a1c3d0fe778874: Status 404 returned error can't find the container with id 069b3af5291ef77dbd2dad03b1fc781133a5432bf0ced48459a1c3d0fe778874
Jul 14 18:08:29.674832 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:29.674792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt" event={"ID":"bb706b6c-f43d-4b35-a511-1d9a9221b192","Type":"ContainerStarted","Data":"e8ae17bd543b3829f4912bd7364532b1980a4da2bd69a494cd9693ce7c3b4b4e"}
Jul 14 18:08:29.675260 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:29.674840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt" event={"ID":"bb706b6c-f43d-4b35-a511-1d9a9221b192","Type":"ContainerStarted","Data":"069b3af5291ef77dbd2dad03b1fc781133a5432bf0ced48459a1c3d0fe778874"}
Jul 14 18:08:29.708428 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:29.708371    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt" podStartSLOduration=1.708355283 podStartE2EDuration="1.708355283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:08:29.706651335 +0000 UTC m=+2166.423045028" watchObservedRunningTime="2026-07-14 18:08:29.708355283 +0000 UTC m=+2166.424748974"
Jul 14 18:08:31.806673 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:31.806642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:31.955175 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:31.955144    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:08:32.415324 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:32.415283    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:08:38.764391 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:38.764352    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:38.764863 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:38.764405    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:38.777062 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:38.777037    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:08:39.745981 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:08:39.745951    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:09:10.280394 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.280361    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"]
Jul 14 18:09:10.284114 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.284097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.287703 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.287669    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9c28f2\"/\"test-model-9c28f2-kserve-self-signed-certs\""
Jul 14 18:09:10.287842 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.287709    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9c28f2\"/\"openshift-service-ca.crt\""
Jul 14 18:09:10.287842 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.287748    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9c28f2\"/\"kube-root-ca.crt\""
Jul 14 18:09:10.287842 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.287669    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9c28f2\"/\"default-dockercfg-b595v\""
Jul 14 18:09:10.300306 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.300279    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"]
Jul 14 18:09:10.307406 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.307377    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:09:10.385610 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.385576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daff7fc2-9325-488d-b1a6-f5f843b0816b-tls-certs\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.385610 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.385613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6x5x\" (UniqueName: \"kubernetes.io/projected/daff7fc2-9325-488d-b1a6-f5f843b0816b-kube-api-access-n6x5x\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.385848 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.385694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-home\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.385848 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.385730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-model-cache\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.385848 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.385788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-dshm\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.385848 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.385812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-tmp-dir\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.486912 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.486878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daff7fc2-9325-488d-b1a6-f5f843b0816b-tls-certs\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.486912 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.486923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6x5x\" (UniqueName: \"kubernetes.io/projected/daff7fc2-9325-488d-b1a6-f5f843b0816b-kube-api-access-n6x5x\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.487172 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.487060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-home\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.487172 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.487098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-model-cache\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.487335 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.487171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-dshm\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.487335 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.487257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-tmp-dir\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.487518 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.487501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-home\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.487558 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.487518    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-model-cache\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.487649 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.487633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-tmp-dir\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.489444 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.489422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-dshm\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.489589 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.489571    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daff7fc2-9325-488d-b1a6-f5f843b0816b-tls-certs\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.497270 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.497244    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6x5x\" (UniqueName: \"kubernetes.io/projected/daff7fc2-9325-488d-b1a6-f5f843b0816b-kube-api-access-n6x5x\") pod \"test-model-9c28f2-kserve-cdff4bbc7-6hklx\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") " pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.594008 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.593919    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:10.737571 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.737549    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"]
Jul 14 18:09:10.739484 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:09:10.739452    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaff7fc2_9325_488d_b1a6_f5f843b0816b.slice/crio-0cd334b147b37f14b49398e3c05762a69be95d4153aeb933da2d45006b80e769 WatchSource:0}: Error finding container 0cd334b147b37f14b49398e3c05762a69be95d4153aeb933da2d45006b80e769: Status 404 returned error can't find the container with id 0cd334b147b37f14b49398e3c05762a69be95d4153aeb933da2d45006b80e769
Jul 14 18:09:10.923185 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.923151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx" event={"ID":"daff7fc2-9325-488d-b1a6-f5f843b0816b","Type":"ContainerStarted","Data":"30fe6f90bbb5220c483d180ee9dad1861806b6d963bad44786c2955646197d8c"}
Jul 14 18:09:10.923394 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.923195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx" event={"ID":"daff7fc2-9325-488d-b1a6-f5f843b0816b","Type":"ContainerStarted","Data":"0cd334b147b37f14b49398e3c05762a69be95d4153aeb933da2d45006b80e769"}
Jul 14 18:09:10.948800 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:10.948746    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx" podStartSLOduration=0.94872953 podStartE2EDuration="948.72953ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:09:10.945529604 +0000 UTC m=+2207.661923309" watchObservedRunningTime="2026-07-14 18:09:10.94872953 +0000 UTC m=+2207.665123225"
Jul 14 18:09:15.526656 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:15.526618    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:09:20.595004 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:20.594953    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:20.595004 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:20.595015    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:20.607817 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:20.607787    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:21.002547 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:21.002517    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:09:44.113358 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:44.113271    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:09:53.784335 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:53.784307    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"]
Jul 14 18:09:53.820849 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:53.820818    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:09:54.733707 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:54.733672    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"]
Jul 14 18:09:54.784860 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:54.784830    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:09:55.547985 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:55.547947    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:09:55.816887 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:55.816802    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:09:58.900772 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:09:58.900734    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:10:05.635845 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:05.635812    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"]
Jul 14 18:10:05.636395 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:05.636057    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9" podUID="402f5976-86cc-402a-b0ad-91176a60e45b" containerName="maas-api" containerID="cri-o://0acea30460473503edbc87b85f3ca21f6c49611b3b8a387c6eb3f1efa6b0e732" gracePeriod=30
Jul 14 18:10:05.663926 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:05.663901    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:05.894781 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:05.894722    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:10:06.018403 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.018362    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r46l\" (UniqueName: \"kubernetes.io/projected/402f5976-86cc-402a-b0ad-91176a60e45b-kube-api-access-5r46l\") pod \"402f5976-86cc-402a-b0ad-91176a60e45b\" (UID: \"402f5976-86cc-402a-b0ad-91176a60e45b\") "
Jul 14 18:10:06.018607 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.018434    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402f5976-86cc-402a-b0ad-91176a60e45b-openshift-service-ca\") pod \"402f5976-86cc-402a-b0ad-91176a60e45b\" (UID: \"402f5976-86cc-402a-b0ad-91176a60e45b\") "
Jul 14 18:10:06.018607 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.018468    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402f5976-86cc-402a-b0ad-91176a60e45b-maas-api-tls\") pod \"402f5976-86cc-402a-b0ad-91176a60e45b\" (UID: \"402f5976-86cc-402a-b0ad-91176a60e45b\") "
Jul 14 18:10:06.018849 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.018817    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402f5976-86cc-402a-b0ad-91176a60e45b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "402f5976-86cc-402a-b0ad-91176a60e45b" (UID: "402f5976-86cc-402a-b0ad-91176a60e45b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:10:06.020684 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.020654    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402f5976-86cc-402a-b0ad-91176a60e45b-kube-api-access-5r46l" (OuterVolumeSpecName: "kube-api-access-5r46l") pod "402f5976-86cc-402a-b0ad-91176a60e45b" (UID: "402f5976-86cc-402a-b0ad-91176a60e45b"). InnerVolumeSpecName "kube-api-access-5r46l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:10:06.020789 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.020710    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/402f5976-86cc-402a-b0ad-91176a60e45b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "402f5976-86cc-402a-b0ad-91176a60e45b" (UID: "402f5976-86cc-402a-b0ad-91176a60e45b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:10:06.119974 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.119938    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r46l\" (UniqueName: \"kubernetes.io/projected/402f5976-86cc-402a-b0ad-91176a60e45b-kube-api-access-5r46l\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:06.119974 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.119968    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402f5976-86cc-402a-b0ad-91176a60e45b-openshift-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:06.119974 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.119978    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402f5976-86cc-402a-b0ad-91176a60e45b-maas-api-tls\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:06.268850 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.268810    2578 generic.go:356] "Generic (PLEG): container finished" podID="402f5976-86cc-402a-b0ad-91176a60e45b" containerID="0acea30460473503edbc87b85f3ca21f6c49611b3b8a387c6eb3f1efa6b0e732" exitCode=0
Jul 14 18:10:06.269052 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.268863    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"
Jul 14 18:10:06.269052 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.268868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9" event={"ID":"402f5976-86cc-402a-b0ad-91176a60e45b","Type":"ContainerDied","Data":"0acea30460473503edbc87b85f3ca21f6c49611b3b8a387c6eb3f1efa6b0e732"}
Jul 14 18:10:06.269052 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.268995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9" event={"ID":"402f5976-86cc-402a-b0ad-91176a60e45b","Type":"ContainerDied","Data":"d8c1dc530e6b899cb60a3b452cf7132b5739404c0913af04ba0ed2e07f6c0142"}
Jul 14 18:10:06.269052 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.269027    2578 scope.go:117] "RemoveContainer" containerID="0acea30460473503edbc87b85f3ca21f6c49611b3b8a387c6eb3f1efa6b0e732"
Jul 14 18:10:06.279180 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.279160    2578 scope.go:117] "RemoveContainer" containerID="0acea30460473503edbc87b85f3ca21f6c49611b3b8a387c6eb3f1efa6b0e732"
Jul 14 18:10:06.279488 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:10:06.279468    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acea30460473503edbc87b85f3ca21f6c49611b3b8a387c6eb3f1efa6b0e732\": container with ID starting with 0acea30460473503edbc87b85f3ca21f6c49611b3b8a387c6eb3f1efa6b0e732 not found: ID does not exist" containerID="0acea30460473503edbc87b85f3ca21f6c49611b3b8a387c6eb3f1efa6b0e732"
Jul 14 18:10:06.279546 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.279496    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acea30460473503edbc87b85f3ca21f6c49611b3b8a387c6eb3f1efa6b0e732"} err="failed to get container status \"0acea30460473503edbc87b85f3ca21f6c49611b3b8a387c6eb3f1efa6b0e732\": rpc error: code = NotFound desc = could not find container \"0acea30460473503edbc87b85f3ca21f6c49611b3b8a387c6eb3f1efa6b0e732\": container with ID starting with 0acea30460473503edbc87b85f3ca21f6c49611b3b8a387c6eb3f1efa6b0e732 not found: ID does not exist"
Jul 14 18:10:06.295269 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.295188    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"]
Jul 14 18:10:06.297681 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:06.297663    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae8cb8-8577c8bd79-pfzt9"]
Jul 14 18:10:07.783307 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:07.783275    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="402f5976-86cc-402a-b0ad-91176a60e45b" path="/var/lib/kubelet/pods/402f5976-86cc-402a-b0ad-91176a60e45b/volumes"
Jul 14 18:10:08.804705 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:08.804665    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt" podUID="bb706b6c-f43d-4b35-a511-1d9a9221b192" containerName="main" containerID="cri-o://e8ae17bd543b3829f4912bd7364532b1980a4da2bd69a494cd9693ce7c3b4b4e" gracePeriod=45
Jul 14 18:10:09.070639 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.070615    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:10:09.147421 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.147392    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-tmp-dir\") pod \"bb706b6c-f43d-4b35-a511-1d9a9221b192\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") "
Jul 14 18:10:09.147608 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.147441    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh2dl\" (UniqueName: \"kubernetes.io/projected/bb706b6c-f43d-4b35-a511-1d9a9221b192-kube-api-access-sh2dl\") pod \"bb706b6c-f43d-4b35-a511-1d9a9221b192\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") "
Jul 14 18:10:09.147608 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.147481    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-model-cache\") pod \"bb706b6c-f43d-4b35-a511-1d9a9221b192\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") "
Jul 14 18:10:09.147608 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.147539    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-home\") pod \"bb706b6c-f43d-4b35-a511-1d9a9221b192\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") "
Jul 14 18:10:09.147608 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.147574    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb706b6c-f43d-4b35-a511-1d9a9221b192-tls-certs\") pod \"bb706b6c-f43d-4b35-a511-1d9a9221b192\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") "
Jul 14 18:10:09.147884 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.147610    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-dshm\") pod \"bb706b6c-f43d-4b35-a511-1d9a9221b192\" (UID: \"bb706b6c-f43d-4b35-a511-1d9a9221b192\") "
Jul 14 18:10:09.147884 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.147690    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb706b6c-f43d-4b35-a511-1d9a9221b192" (UID: "bb706b6c-f43d-4b35-a511-1d9a9221b192"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:10:09.147884 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.147801    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb706b6c-f43d-4b35-a511-1d9a9221b192" (UID: "bb706b6c-f43d-4b35-a511-1d9a9221b192"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:10:09.147884 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.147816    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-home" (OuterVolumeSpecName: "home") pod "bb706b6c-f43d-4b35-a511-1d9a9221b192" (UID: "bb706b6c-f43d-4b35-a511-1d9a9221b192"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:10:09.148080 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.147969    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:09.148080 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.147987    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:09.148080 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.148000    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:09.149920 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.149895    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb706b6c-f43d-4b35-a511-1d9a9221b192-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb706b6c-f43d-4b35-a511-1d9a9221b192" (UID: "bb706b6c-f43d-4b35-a511-1d9a9221b192"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:10:09.150041 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.149954    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb706b6c-f43d-4b35-a511-1d9a9221b192-kube-api-access-sh2dl" (OuterVolumeSpecName: "kube-api-access-sh2dl") pod "bb706b6c-f43d-4b35-a511-1d9a9221b192" (UID: "bb706b6c-f43d-4b35-a511-1d9a9221b192"). InnerVolumeSpecName "kube-api-access-sh2dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:10:09.150041 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.149957    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-dshm" (OuterVolumeSpecName: "dshm") pod "bb706b6c-f43d-4b35-a511-1d9a9221b192" (UID: "bb706b6c-f43d-4b35-a511-1d9a9221b192"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:10:09.248699 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.248659    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb706b6c-f43d-4b35-a511-1d9a9221b192-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:09.248699 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.248689    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb706b6c-f43d-4b35-a511-1d9a9221b192-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:09.248699 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.248704    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh2dl\" (UniqueName: \"kubernetes.io/projected/bb706b6c-f43d-4b35-a511-1d9a9221b192-kube-api-access-sh2dl\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:09.289521 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.289489    2578 generic.go:356] "Generic (PLEG): container finished" podID="bb706b6c-f43d-4b35-a511-1d9a9221b192" containerID="e8ae17bd543b3829f4912bd7364532b1980a4da2bd69a494cd9693ce7c3b4b4e" exitCode=0
Jul 14 18:10:09.289668 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.289535    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"
Jul 14 18:10:09.289668 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.289581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt" event={"ID":"bb706b6c-f43d-4b35-a511-1d9a9221b192","Type":"ContainerDied","Data":"e8ae17bd543b3829f4912bd7364532b1980a4da2bd69a494cd9693ce7c3b4b4e"}
Jul 14 18:10:09.289668 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.289625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt" event={"ID":"bb706b6c-f43d-4b35-a511-1d9a9221b192","Type":"ContainerDied","Data":"069b3af5291ef77dbd2dad03b1fc781133a5432bf0ced48459a1c3d0fe778874"}
Jul 14 18:10:09.289668 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.289646    2578 scope.go:117] "RemoveContainer" containerID="e8ae17bd543b3829f4912bd7364532b1980a4da2bd69a494cd9693ce7c3b4b4e"
Jul 14 18:10:09.299363 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.299343    2578 scope.go:117] "RemoveContainer" containerID="e8ae17bd543b3829f4912bd7364532b1980a4da2bd69a494cd9693ce7c3b4b4e"
Jul 14 18:10:09.299626 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:10:09.299608    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ae17bd543b3829f4912bd7364532b1980a4da2bd69a494cd9693ce7c3b4b4e\": container with ID starting with e8ae17bd543b3829f4912bd7364532b1980a4da2bd69a494cd9693ce7c3b4b4e not found: ID does not exist" containerID="e8ae17bd543b3829f4912bd7364532b1980a4da2bd69a494cd9693ce7c3b4b4e"
Jul 14 18:10:09.299670 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.299637    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ae17bd543b3829f4912bd7364532b1980a4da2bd69a494cd9693ce7c3b4b4e"} err="failed to get container status \"e8ae17bd543b3829f4912bd7364532b1980a4da2bd69a494cd9693ce7c3b4b4e\": rpc error: code = NotFound desc = could not find container \"e8ae17bd543b3829f4912bd7364532b1980a4da2bd69a494cd9693ce7c3b4b4e\": container with ID starting with e8ae17bd543b3829f4912bd7364532b1980a4da2bd69a494cd9693ce7c3b4b4e not found: ID does not exist"
Jul 14 18:10:09.314570 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.314499    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"]
Jul 14 18:10:09.324940 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.324905    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ae8cb8/test-model-ae8cb8-kserve-98c4f94dc-c7tnt"]
Jul 14 18:10:09.754454 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.754406    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx" podUID="daff7fc2-9325-488d-b1a6-f5f843b0816b" containerName="main" containerID="cri-o://30fe6f90bbb5220c483d180ee9dad1861806b6d963bad44786c2955646197d8c" gracePeriod=45
Jul 14 18:10:09.783111 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.783084    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb706b6c-f43d-4b35-a511-1d9a9221b192" path="/var/lib/kubelet/pods/bb706b6c-f43d-4b35-a511-1d9a9221b192/volumes"
Jul 14 18:10:09.986799 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:09.986761    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:10.014451 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.014390    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:10:10.157749 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.157714    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6x5x\" (UniqueName: \"kubernetes.io/projected/daff7fc2-9325-488d-b1a6-f5f843b0816b-kube-api-access-n6x5x\") pod \"daff7fc2-9325-488d-b1a6-f5f843b0816b\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") "
Jul 14 18:10:10.157940 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.157780    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daff7fc2-9325-488d-b1a6-f5f843b0816b-tls-certs\") pod \"daff7fc2-9325-488d-b1a6-f5f843b0816b\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") "
Jul 14 18:10:10.157940 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.157826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-model-cache\") pod \"daff7fc2-9325-488d-b1a6-f5f843b0816b\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") "
Jul 14 18:10:10.157940 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.157841    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-dshm\") pod \"daff7fc2-9325-488d-b1a6-f5f843b0816b\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") "
Jul 14 18:10:10.157940 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.157877    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-home\") pod \"daff7fc2-9325-488d-b1a6-f5f843b0816b\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") "
Jul 14 18:10:10.157940 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.157919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-tmp-dir\") pod \"daff7fc2-9325-488d-b1a6-f5f843b0816b\" (UID: \"daff7fc2-9325-488d-b1a6-f5f843b0816b\") "
Jul 14 18:10:10.158201 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.158101    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-model-cache" (OuterVolumeSpecName: "model-cache") pod "daff7fc2-9325-488d-b1a6-f5f843b0816b" (UID: "daff7fc2-9325-488d-b1a6-f5f843b0816b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:10:10.158201 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.158188    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-home" (OuterVolumeSpecName: "home") pod "daff7fc2-9325-488d-b1a6-f5f843b0816b" (UID: "daff7fc2-9325-488d-b1a6-f5f843b0816b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:10:10.158309 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.158281    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "daff7fc2-9325-488d-b1a6-f5f843b0816b" (UID: "daff7fc2-9325-488d-b1a6-f5f843b0816b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:10:10.160112 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.160091    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-dshm" (OuterVolumeSpecName: "dshm") pod "daff7fc2-9325-488d-b1a6-f5f843b0816b" (UID: "daff7fc2-9325-488d-b1a6-f5f843b0816b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:10:10.160493 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.160474    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daff7fc2-9325-488d-b1a6-f5f843b0816b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "daff7fc2-9325-488d-b1a6-f5f843b0816b" (UID: "daff7fc2-9325-488d-b1a6-f5f843b0816b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:10:10.160567 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.160548    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daff7fc2-9325-488d-b1a6-f5f843b0816b-kube-api-access-n6x5x" (OuterVolumeSpecName: "kube-api-access-n6x5x") pod "daff7fc2-9325-488d-b1a6-f5f843b0816b" (UID: "daff7fc2-9325-488d-b1a6-f5f843b0816b"). InnerVolumeSpecName "kube-api-access-n6x5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:10:10.258934 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.258891    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:10.258934 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.258929    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:10.258934 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.258940    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6x5x\" (UniqueName: \"kubernetes.io/projected/daff7fc2-9325-488d-b1a6-f5f843b0816b-kube-api-access-n6x5x\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:10.259174 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.258949    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daff7fc2-9325-488d-b1a6-f5f843b0816b-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:10.259174 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.258959    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:10.259174 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.258967    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daff7fc2-9325-488d-b1a6-f5f843b0816b-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:10.297346 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.297255    2578 generic.go:356] "Generic (PLEG): container finished" podID="daff7fc2-9325-488d-b1a6-f5f843b0816b" containerID="30fe6f90bbb5220c483d180ee9dad1861806b6d963bad44786c2955646197d8c" exitCode=0
Jul 14 18:10:10.297346 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.297302    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"
Jul 14 18:10:10.297346 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.297328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx" event={"ID":"daff7fc2-9325-488d-b1a6-f5f843b0816b","Type":"ContainerDied","Data":"30fe6f90bbb5220c483d180ee9dad1861806b6d963bad44786c2955646197d8c"}
Jul 14 18:10:10.297597 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.297366    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx" event={"ID":"daff7fc2-9325-488d-b1a6-f5f843b0816b","Type":"ContainerDied","Data":"0cd334b147b37f14b49398e3c05762a69be95d4153aeb933da2d45006b80e769"}
Jul 14 18:10:10.297597 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.297387    2578 scope.go:117] "RemoveContainer" containerID="30fe6f90bbb5220c483d180ee9dad1861806b6d963bad44786c2955646197d8c"
Jul 14 18:10:10.307132 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.307114    2578 scope.go:117] "RemoveContainer" containerID="30fe6f90bbb5220c483d180ee9dad1861806b6d963bad44786c2955646197d8c"
Jul 14 18:10:10.307396 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:10:10.307379    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30fe6f90bbb5220c483d180ee9dad1861806b6d963bad44786c2955646197d8c\": container with ID starting with 30fe6f90bbb5220c483d180ee9dad1861806b6d963bad44786c2955646197d8c not found: ID does not exist" containerID="30fe6f90bbb5220c483d180ee9dad1861806b6d963bad44786c2955646197d8c"
Jul 14 18:10:10.307454 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.307402    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30fe6f90bbb5220c483d180ee9dad1861806b6d963bad44786c2955646197d8c"} err="failed to get container status \"30fe6f90bbb5220c483d180ee9dad1861806b6d963bad44786c2955646197d8c\": rpc error: code = NotFound desc = could not find container \"30fe6f90bbb5220c483d180ee9dad1861806b6d963bad44786c2955646197d8c\": container with ID starting with 30fe6f90bbb5220c483d180ee9dad1861806b6d963bad44786c2955646197d8c not found: ID does not exist"
Jul 14 18:10:10.324850 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.324825    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"]
Jul 14 18:10:10.332016 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:10.331994    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9c28f2/test-model-9c28f2-kserve-cdff4bbc7-6hklx"]
Jul 14 18:10:11.312907 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:11.312869    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:10:11.783008 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:11.782979    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daff7fc2-9325-488d-b1a6-f5f843b0816b" path="/var/lib/kubelet/pods/daff7fc2-9325-488d-b1a6-f5f843b0816b/volumes"
Jul 14 18:10:17.076404 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:17.076366    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:17.188495 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:17.188463    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:20.711272 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:20.711233    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:20.972917 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:20.972833    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:21.102818 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.102779    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg"]
Jul 14 18:10:21.103247 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.103230    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="402f5976-86cc-402a-b0ad-91176a60e45b" containerName="maas-api"
Jul 14 18:10:21.103296 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.103252    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="402f5976-86cc-402a-b0ad-91176a60e45b" containerName="maas-api"
Jul 14 18:10:21.103296 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.103265    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daff7fc2-9325-488d-b1a6-f5f843b0816b" containerName="main"
Jul 14 18:10:21.103296 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.103271    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="daff7fc2-9325-488d-b1a6-f5f843b0816b" containerName="main"
Jul 14 18:10:21.103296 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.103281    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb706b6c-f43d-4b35-a511-1d9a9221b192" containerName="main"
Jul 14 18:10:21.103296 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.103286    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb706b6c-f43d-4b35-a511-1d9a9221b192" containerName="main"
Jul 14 18:10:21.103454 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.103353    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="402f5976-86cc-402a-b0ad-91176a60e45b" containerName="maas-api"
Jul 14 18:10:21.103454 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.103363    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="daff7fc2-9325-488d-b1a6-f5f843b0816b" containerName="main"
Jul 14 18:10:21.103454 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.103370    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb706b6c-f43d-4b35-a511-1d9a9221b192" containerName="main"
Jul 14 18:10:21.107935 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.107917    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg"
Jul 14 18:10:21.112534 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.112505    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pd7w9\""
Jul 14 18:10:21.119788 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.119759    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg"]
Jul 14 18:10:21.259323 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.259240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a512ffcd-7e14-492e-8aa2-718e31db270b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg\" (UID: \"a512ffcd-7e14-492e-8aa2-718e31db270b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg"
Jul 14 18:10:21.360058 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.360022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a512ffcd-7e14-492e-8aa2-718e31db270b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg\" (UID: \"a512ffcd-7e14-492e-8aa2-718e31db270b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg"
Jul 14 18:10:21.360642 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.360623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a512ffcd-7e14-492e-8aa2-718e31db270b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg\" (UID: \"a512ffcd-7e14-492e-8aa2-718e31db270b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg"
Jul 14 18:10:21.418103 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.418069    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg"
Jul 14 18:10:21.586407 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:21.586376    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg"]
Jul 14 18:10:21.587698 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:10:21.587673    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda512ffcd_7e14_492e_8aa2_718e31db270b.slice/crio-5e697af26d182a501a10ae19c0f92e7a7e818fcedda32576963bcdf305a84267 WatchSource:0}: Error finding container 5e697af26d182a501a10ae19c0f92e7a7e818fcedda32576963bcdf305a84267: Status 404 returned error can't find the container with id 5e697af26d182a501a10ae19c0f92e7a7e818fcedda32576963bcdf305a84267
Jul 14 18:10:22.010265 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:22.010230    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:22.110259 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:22.110227    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:22.374478 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:22.374436    2578 generic.go:356] "Generic (PLEG): container finished" podID="a512ffcd-7e14-492e-8aa2-718e31db270b" containerID="c71221e2cca141610427659c9ef3dc4ab4fcddcd72dd9cfdd7e60b5cd0e21bb6" exitCode=0
Jul 14 18:10:22.374744 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:22.374711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg" event={"ID":"a512ffcd-7e14-492e-8aa2-718e31db270b","Type":"ContainerDied","Data":"c71221e2cca141610427659c9ef3dc4ab4fcddcd72dd9cfdd7e60b5cd0e21bb6"}
Jul 14 18:10:22.375099 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:22.375051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg" event={"ID":"a512ffcd-7e14-492e-8aa2-718e31db270b","Type":"ContainerStarted","Data":"5e697af26d182a501a10ae19c0f92e7a7e818fcedda32576963bcdf305a84267"}
Jul 14 18:10:23.517453 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:23.517423    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg"
Jul 14 18:10:23.680770 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:23.680734    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a512ffcd-7e14-492e-8aa2-718e31db270b-maas-api-service-ca\") pod \"a512ffcd-7e14-492e-8aa2-718e31db270b\" (UID: \"a512ffcd-7e14-492e-8aa2-718e31db270b\") "
Jul 14 18:10:23.681051 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:23.681027    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a512ffcd-7e14-492e-8aa2-718e31db270b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a512ffcd-7e14-492e-8aa2-718e31db270b" (UID: "a512ffcd-7e14-492e-8aa2-718e31db270b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:10:23.781695 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:23.781655    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a512ffcd-7e14-492e-8aa2-718e31db270b-maas-api-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:24.387747 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:24.387716    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg"
Jul 14 18:10:24.387922 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:24.387750    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg" event={"ID":"a512ffcd-7e14-492e-8aa2-718e31db270b","Type":"ContainerDied","Data":"5e697af26d182a501a10ae19c0f92e7a7e818fcedda32576963bcdf305a84267"}
Jul 14 18:10:24.387922 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:24.387797    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e697af26d182a501a10ae19c0f92e7a7e818fcedda32576963bcdf305a84267"
Jul 14 18:10:27.545600 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:27.545563    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:27.647362 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:27.647318    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:32.062731 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:32.062694    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:32.160025 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:32.159984    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:32.378512 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:32.378429    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"]
Jul 14 18:10:32.378668 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:32.378604    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t" podUID="d23e6a4a-47cc-423e-ba42-5330dec64e88" containerName="maas-api" containerID="cri-o://733727e25516f4c478958e4532151e5cc37244b8bfd24c451d480eb5edc67c79" gracePeriod=30
Jul 14 18:10:32.419635 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:32.419603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:32.624349 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:32.624325    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:10:32.654371 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:32.654294    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d23e6a4a-47cc-423e-ba42-5330dec64e88-openshift-service-ca\") pod \"d23e6a4a-47cc-423e-ba42-5330dec64e88\" (UID: \"d23e6a4a-47cc-423e-ba42-5330dec64e88\") "
Jul 14 18:10:32.654371 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:32.654330    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n5xd\" (UniqueName: \"kubernetes.io/projected/d23e6a4a-47cc-423e-ba42-5330dec64e88-kube-api-access-5n5xd\") pod \"d23e6a4a-47cc-423e-ba42-5330dec64e88\" (UID: \"d23e6a4a-47cc-423e-ba42-5330dec64e88\") "
Jul 14 18:10:32.654371 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:32.654361    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d23e6a4a-47cc-423e-ba42-5330dec64e88-maas-api-tls\") pod \"d23e6a4a-47cc-423e-ba42-5330dec64e88\" (UID: \"d23e6a4a-47cc-423e-ba42-5330dec64e88\") "
Jul 14 18:10:32.654715 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:32.654694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d23e6a4a-47cc-423e-ba42-5330dec64e88-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d23e6a4a-47cc-423e-ba42-5330dec64e88" (UID: "d23e6a4a-47cc-423e-ba42-5330dec64e88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:10:32.657052 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:32.656889    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23e6a4a-47cc-423e-ba42-5330dec64e88-kube-api-access-5n5xd" (OuterVolumeSpecName: "kube-api-access-5n5xd") pod "d23e6a4a-47cc-423e-ba42-5330dec64e88" (UID: "d23e6a4a-47cc-423e-ba42-5330dec64e88"). InnerVolumeSpecName "kube-api-access-5n5xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:10:32.657052 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:32.656983    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23e6a4a-47cc-423e-ba42-5330dec64e88-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d23e6a4a-47cc-423e-ba42-5330dec64e88" (UID: "d23e6a4a-47cc-423e-ba42-5330dec64e88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:10:32.755147 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:32.755111    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d23e6a4a-47cc-423e-ba42-5330dec64e88-openshift-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:32.755147 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:32.755139    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n5xd\" (UniqueName: \"kubernetes.io/projected/d23e6a4a-47cc-423e-ba42-5330dec64e88-kube-api-access-5n5xd\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:32.755147 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:32.755149    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d23e6a4a-47cc-423e-ba42-5330dec64e88-maas-api-tls\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:10:33.439862 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:33.439826    2578 generic.go:356] "Generic (PLEG): container finished" podID="d23e6a4a-47cc-423e-ba42-5330dec64e88" containerID="733727e25516f4c478958e4532151e5cc37244b8bfd24c451d480eb5edc67c79" exitCode=0
Jul 14 18:10:33.440281 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:33.439875    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"
Jul 14 18:10:33.440281 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:33.439916    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t" event={"ID":"d23e6a4a-47cc-423e-ba42-5330dec64e88","Type":"ContainerDied","Data":"733727e25516f4c478958e4532151e5cc37244b8bfd24c451d480eb5edc67c79"}
Jul 14 18:10:33.440281 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:33.439970    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t" event={"ID":"d23e6a4a-47cc-423e-ba42-5330dec64e88","Type":"ContainerDied","Data":"3230944d4836affc9e389f86121e4141915dd798fdee28b7ad34b029653034df"}
Jul 14 18:10:33.440281 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:33.439992    2578 scope.go:117] "RemoveContainer" containerID="733727e25516f4c478958e4532151e5cc37244b8bfd24c451d480eb5edc67c79"
Jul 14 18:10:33.452304 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:33.452283    2578 scope.go:117] "RemoveContainer" containerID="733727e25516f4c478958e4532151e5cc37244b8bfd24c451d480eb5edc67c79"
Jul 14 18:10:33.452609 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:10:33.452583    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733727e25516f4c478958e4532151e5cc37244b8bfd24c451d480eb5edc67c79\": container with ID starting with 733727e25516f4c478958e4532151e5cc37244b8bfd24c451d480eb5edc67c79 not found: ID does not exist" containerID="733727e25516f4c478958e4532151e5cc37244b8bfd24c451d480eb5edc67c79"
Jul 14 18:10:33.452671 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:33.452622    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733727e25516f4c478958e4532151e5cc37244b8bfd24c451d480eb5edc67c79"} err="failed to get container status \"733727e25516f4c478958e4532151e5cc37244b8bfd24c451d480eb5edc67c79\": rpc error: code = NotFound desc = could not find container \"733727e25516f4c478958e4532151e5cc37244b8bfd24c451d480eb5edc67c79\": container with ID starting with 733727e25516f4c478958e4532151e5cc37244b8bfd24c451d480eb5edc67c79 not found: ID does not exist"
Jul 14 18:10:33.478263 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:33.478232    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"]
Jul 14 18:10:33.486552 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:33.486524    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t"]
Jul 14 18:10:33.785049 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:33.784964    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d23e6a4a-47cc-423e-ba42-5330dec64e88" path="/var/lib/kubelet/pods/d23e6a4a-47cc-423e-ba42-5330dec64e88/volumes"
Jul 14 18:10:35.140278 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:35.140248    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:10:35.567509 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:35.567446    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c28f2-6fd87f684c-5dr2t" podUID="d23e6a4a-47cc-423e-ba42-5330dec64e88" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.67:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 14 18:10:36.267320 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:36.267285    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:40.265940 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:40.265907    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:43.084168 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:43.084136    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:43.183687 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:43.183649    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:47.121092 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:47.121052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:47.346785 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:47.346749    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:47.636540 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:47.636508    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:48.038384 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:48.038355    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:48.283196 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:48.283164    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:51.635097 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.635064    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:51.838631 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.838598    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:51.965536 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.965505    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"]
Jul 14 18:10:51.965909 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.965897    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a512ffcd-7e14-492e-8aa2-718e31db270b" containerName="revoke-keys"
Jul 14 18:10:51.965968 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.965911    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a512ffcd-7e14-492e-8aa2-718e31db270b" containerName="revoke-keys"
Jul 14 18:10:51.965968 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.965925    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23e6a4a-47cc-423e-ba42-5330dec64e88" containerName="maas-api"
Jul 14 18:10:51.965968 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.965930    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23e6a4a-47cc-423e-ba42-5330dec64e88" containerName="maas-api"
Jul 14 18:10:51.966074 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.965989    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23e6a4a-47cc-423e-ba42-5330dec64e88" containerName="maas-api"
Jul 14 18:10:51.966074 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.966001    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a512ffcd-7e14-492e-8aa2-718e31db270b" containerName="revoke-keys"
Jul 14 18:10:51.970571 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.970555    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"
Jul 14 18:10:51.974160 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.974140    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5a43f9-serving-cert\""
Jul 14 18:10:51.974399 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.974385    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xxs8g\""
Jul 14 18:10:51.974891 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.974868    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 18:10:51.975295 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.975278    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 18:10:51.990707 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.990681    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:51.996165 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:51.996145    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"]
Jul 14 18:10:52.012481 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:52.012448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-openshift-service-ca\") pod \"maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls\" (UID: \"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"
Jul 14 18:10:52.012654 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:52.012506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc4g6\" (UniqueName: \"kubernetes.io/projected/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-kube-api-access-mc4g6\") pod \"maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls\" (UID: \"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"
Jul 14 18:10:52.012654 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:52.012612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-maas-api-tls\") pod \"maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls\" (UID: \"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"
Jul 14 18:10:52.113550 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:52.113520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-openshift-service-ca\") pod \"maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls\" (UID: \"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"
Jul 14 18:10:52.113745 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:52.113558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc4g6\" (UniqueName: \"kubernetes.io/projected/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-kube-api-access-mc4g6\") pod \"maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls\" (UID: \"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"
Jul 14 18:10:52.113745 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:52.113620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-maas-api-tls\") pod \"maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls\" (UID: \"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"
Jul 14 18:10:52.114188 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:52.114167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-openshift-service-ca\") pod \"maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls\" (UID: \"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"
Jul 14 18:10:52.116286 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:52.116262    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-maas-api-tls\") pod \"maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls\" (UID: \"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"
Jul 14 18:10:52.129434 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:52.129400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc4g6\" (UniqueName: \"kubernetes.io/projected/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-kube-api-access-mc4g6\") pod \"maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls\" (UID: \"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"
Jul 14 18:10:52.190120 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:52.190084    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:52.280324 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:52.280251    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"
Jul 14 18:10:52.446256 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:52.446225    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"]
Jul 14 18:10:52.447427 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:10:52.447401    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcb60bb_ee0d_4990_9bc4_eee79c7f1b5c.slice/crio-30192cbe6bccb2cd68998682864d19346d9ff056f68b824d33e14a83bd668fcf WatchSource:0}: Error finding container 30192cbe6bccb2cd68998682864d19346d9ff056f68b824d33e14a83bd668fcf: Status 404 returned error can't find the container with id 30192cbe6bccb2cd68998682864d19346d9ff056f68b824d33e14a83bd668fcf
Jul 14 18:10:52.448735 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:52.448713    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 18:10:52.546473 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:52.546393    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls" event={"ID":"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c","Type":"ContainerStarted","Data":"30192cbe6bccb2cd68998682864d19346d9ff056f68b824d33e14a83bd668fcf"}
Jul 14 18:10:52.856307 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:52.856227    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:53.070825 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:53.070798    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:53.553666 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:53.553630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls" event={"ID":"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c","Type":"ContainerStarted","Data":"286668344c09cd3eb4d034b7aed61498d3111a94e7cb79842a3f9bbcfa6efe1d"}
Jul 14 18:10:53.553840 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:53.553708    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"
Jul 14 18:10:53.587307 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:53.587255    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls" podStartSLOduration=1.982828514 podStartE2EDuration="2.587234547s" totalImagesPullingTime="604.406033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:53.585810084 +0000 UTC m=+2310.302203779" watchObservedRunningTime="2026-07-14 18:10:53.587234547 +0000 UTC m=+2310.303628238"
Jul 14 18:10:55.930636 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:55.930599    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:56.205683 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:56.205605    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:10:56.486184 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:56.486077    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:57.055750 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:57.055699    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:57.819028 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:57.818994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:59.459595 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:59.459559    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:10:59.522605 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:59.522575    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"]
Jul 14 18:10:59.526085 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:59.526066    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:10:59.535676 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:59.535658    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e9d840-serving-cert\""
Jul 14 18:10:59.548993 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:59.548966    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"]
Jul 14 18:10:59.565029 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:59.565004    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"
Jul 14 18:10:59.582957 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:59.582930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-maas-api-tls\") pod \"maas-api-e2e-api-b-e9d840-6f6577f6b-v985n\" (UID: \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:10:59.583085 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:59.582999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-openshift-service-ca\") pod \"maas-api-e2e-api-b-e9d840-6f6577f6b-v985n\" (UID: \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:10:59.583085 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:59.583023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssplb\" (UniqueName: \"kubernetes.io/projected/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-kube-api-access-ssplb\") pod \"maas-api-e2e-api-b-e9d840-6f6577f6b-v985n\" (UID: \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:10:59.684500 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:59.684470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-maas-api-tls\") pod \"maas-api-e2e-api-b-e9d840-6f6577f6b-v985n\" (UID: \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:10:59.684679 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:59.684656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-openshift-service-ca\") pod \"maas-api-e2e-api-b-e9d840-6f6577f6b-v985n\" (UID: \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:10:59.684755 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:59.684686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssplb\" (UniqueName: \"kubernetes.io/projected/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-kube-api-access-ssplb\") pod \"maas-api-e2e-api-b-e9d840-6f6577f6b-v985n\" (UID: \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:10:59.684755 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:10:59.684698    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-serving-cert: secret "maas-api-e2e-api-b-e9d840-serving-cert" not found
Jul 14 18:10:59.684847 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:10:59.684763    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-maas-api-tls podName:cc58c63d-266a-4e5f-a51c-01a8d10a32bf nodeName:}" failed. No retries permitted until 2026-07-14 18:11:00.184739956 +0000 UTC m=+2316.901133625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-maas-api-tls") pod "maas-api-e2e-api-b-e9d840-6f6577f6b-v985n" (UID: "cc58c63d-266a-4e5f-a51c-01a8d10a32bf") : secret "maas-api-e2e-api-b-e9d840-serving-cert" not found
Jul 14 18:10:59.685329 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:59.685309    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-openshift-service-ca\") pod \"maas-api-e2e-api-b-e9d840-6f6577f6b-v985n\" (UID: \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:10:59.698982 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:59.698951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssplb\" (UniqueName: \"kubernetes.io/projected/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-kube-api-access-ssplb\") pod \"maas-api-e2e-api-b-e9d840-6f6577f6b-v985n\" (UID: \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:10:59.964512 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:10:59.964473    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:00.109983 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:00.109949    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:00.189982 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:00.189950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-maas-api-tls\") pod \"maas-api-e2e-api-b-e9d840-6f6577f6b-v985n\" (UID: \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:11:00.192551 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:00.192530    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-maas-api-tls\") pod \"maas-api-e2e-api-b-e9d840-6f6577f6b-v985n\" (UID: \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:11:00.298468 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:00.298378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:00.436377 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:00.436338    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:11:00.814762 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:00.814735    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"]
Jul 14 18:11:00.816223 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:11:00.816182    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc58c63d_266a_4e5f_a51c_01a8d10a32bf.slice/crio-3cba9e8d0e759dc57d4358474cb49a40826e937fcba5ce32fd5139fedcf91695 WatchSource:0}: Error finding container 3cba9e8d0e759dc57d4358474cb49a40826e937fcba5ce32fd5139fedcf91695: Status 404 returned error can't find the container with id 3cba9e8d0e759dc57d4358474cb49a40826e937fcba5ce32fd5139fedcf91695
Jul 14 18:11:01.128106 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:01.128019    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:01.601085 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:01.601052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n" event={"ID":"cc58c63d-266a-4e5f-a51c-01a8d10a32bf","Type":"ContainerStarted","Data":"3cba9e8d0e759dc57d4358474cb49a40826e937fcba5ce32fd5139fedcf91695"}
Jul 14 18:11:02.608931 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:02.608893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n" event={"ID":"cc58c63d-266a-4e5f-a51c-01a8d10a32bf","Type":"ContainerStarted","Data":"b9fc1f2ff0ddee253afe5399f55353dc1de187da985a98a33c19df765f77d3b2"}
Jul 14 18:11:02.609395 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:02.609076    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:11:02.635069 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:02.635016    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n" podStartSLOduration=2.499040582 podStartE2EDuration="3.635000096s" totalImagesPullingTime="1.135959514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:02.631580345 +0000 UTC m=+2319.347974055" watchObservedRunningTime="2026-07-14 18:11:02.635000096 +0000 UTC m=+2319.351393787"
Jul 14 18:11:03.940968 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:03.940925    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:08.620458 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:08.620430    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:11:10.815626 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:10.815590    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:11:12.009029 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:12.008990    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5"]
Jul 14 18:11:12.012828 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:12.012813    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5"
Jul 14 18:11:12.017110 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:12.017089    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pd7w9\""
Jul 14 18:11:12.028366 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:12.028336    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5"]
Jul 14 18:11:12.096922 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:12.096887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dfb536-7f3a-4319-bbd3-ad40d63d7e30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5\" (UID: \"b0dfb536-7f3a-4319-bbd3-ad40d63d7e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5"
Jul 14 18:11:12.197675 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:12.197638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dfb536-7f3a-4319-bbd3-ad40d63d7e30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5\" (UID: \"b0dfb536-7f3a-4319-bbd3-ad40d63d7e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5"
Jul 14 18:11:12.198325 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:12.198307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dfb536-7f3a-4319-bbd3-ad40d63d7e30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5\" (UID: \"b0dfb536-7f3a-4319-bbd3-ad40d63d7e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5"
Jul 14 18:11:12.322757 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:12.322656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5"
Jul 14 18:11:12.480408 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:12.480378    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5"]
Jul 14 18:11:12.481430 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:11:12.481401    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0dfb536_7f3a_4319_bbd3_ad40d63d7e30.slice/crio-e0ee94d7b4f301829866106a5955e83950eb9d4c3997073fba61ec7846347759 WatchSource:0}: Error finding container e0ee94d7b4f301829866106a5955e83950eb9d4c3997073fba61ec7846347759: Status 404 returned error can't find the container with id e0ee94d7b4f301829866106a5955e83950eb9d4c3997073fba61ec7846347759
Jul 14 18:11:12.675708 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:12.675677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5" event={"ID":"b0dfb536-7f3a-4319-bbd3-ad40d63d7e30","Type":"ContainerStarted","Data":"e0ee94d7b4f301829866106a5955e83950eb9d4c3997073fba61ec7846347759"}
Jul 14 18:11:12.697436 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:12.697407    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:12.788415 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:12.788390    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:13.683555 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:13.683521    2578 generic.go:356] "Generic (PLEG): container finished" podID="b0dfb536-7f3a-4319-bbd3-ad40d63d7e30" containerID="aa56f9f5bcf01aca8066ec4d25d8c5caac7f447800e4f4863c28de4137a2d7f4" exitCode=0
Jul 14 18:11:13.683953 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:13.683615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5" event={"ID":"b0dfb536-7f3a-4319-bbd3-ad40d63d7e30","Type":"ContainerDied","Data":"aa56f9f5bcf01aca8066ec4d25d8c5caac7f447800e4f4863c28de4137a2d7f4"}
Jul 14 18:11:14.836828 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:14.836800    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5"
Jul 14 18:11:14.928055 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:14.928019    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dfb536-7f3a-4319-bbd3-ad40d63d7e30-maas-api-service-ca\") pod \"b0dfb536-7f3a-4319-bbd3-ad40d63d7e30\" (UID: \"b0dfb536-7f3a-4319-bbd3-ad40d63d7e30\") "
Jul 14 18:11:14.928419 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:14.928390    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0dfb536-7f3a-4319-bbd3-ad40d63d7e30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b0dfb536-7f3a-4319-bbd3-ad40d63d7e30" (UID: "b0dfb536-7f3a-4319-bbd3-ad40d63d7e30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:11:15.029243 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:15.029133    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dfb536-7f3a-4319-bbd3-ad40d63d7e30-maas-api-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:11:15.697570 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:15.697540    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5"
Jul 14 18:11:15.697775 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:15.697535    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5" event={"ID":"b0dfb536-7f3a-4319-bbd3-ad40d63d7e30","Type":"ContainerDied","Data":"e0ee94d7b4f301829866106a5955e83950eb9d4c3997073fba61ec7846347759"}
Jul 14 18:11:15.697775 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:15.697668    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0ee94d7b4f301829866106a5955e83950eb9d4c3997073fba61ec7846347759"
Jul 14 18:11:21.520444 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:21.520414    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:21.619737 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:21.619671    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:22.434483 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.434446    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"]
Jul 14 18:11:22.434675 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.434654    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls" podUID="bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c" containerName="maas-api" containerID="cri-o://286668344c09cd3eb4d034b7aed61498d3111a94e7cb79842a3f9bbcfa6efe1d" gracePeriod=30
Jul 14 18:11:22.462720 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.462696    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:22.689064 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.689009    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"
Jul 14 18:11:22.698062 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.698041    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc4g6\" (UniqueName: \"kubernetes.io/projected/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-kube-api-access-mc4g6\") pod \"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c\" (UID: \"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c\") "
Jul 14 18:11:22.698132 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.698086    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-maas-api-tls\") pod \"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c\" (UID: \"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c\") "
Jul 14 18:11:22.698189 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.698178    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-openshift-service-ca\") pod \"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c\" (UID: \"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c\") "
Jul 14 18:11:22.698614 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.698590    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c" (UID: "bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:11:22.700306 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.700278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c" (UID: "bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:11:22.700433 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.700414    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-kube-api-access-mc4g6" (OuterVolumeSpecName: "kube-api-access-mc4g6") pod "bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c" (UID: "bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c"). InnerVolumeSpecName "kube-api-access-mc4g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:11:22.746073 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.746041    2578 generic.go:356] "Generic (PLEG): container finished" podID="bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c" containerID="286668344c09cd3eb4d034b7aed61498d3111a94e7cb79842a3f9bbcfa6efe1d" exitCode=0
Jul 14 18:11:22.746286 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.746105    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"
Jul 14 18:11:22.746286 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.746168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls" event={"ID":"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c","Type":"ContainerDied","Data":"286668344c09cd3eb4d034b7aed61498d3111a94e7cb79842a3f9bbcfa6efe1d"}
Jul 14 18:11:22.746286 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.746250    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls" event={"ID":"bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c","Type":"ContainerDied","Data":"30192cbe6bccb2cd68998682864d19346d9ff056f68b824d33e14a83bd668fcf"}
Jul 14 18:11:22.746474 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.746288    2578 scope.go:117] "RemoveContainer" containerID="286668344c09cd3eb4d034b7aed61498d3111a94e7cb79842a3f9bbcfa6efe1d"
Jul 14 18:11:22.756044 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.756028    2578 scope.go:117] "RemoveContainer" containerID="286668344c09cd3eb4d034b7aed61498d3111a94e7cb79842a3f9bbcfa6efe1d"
Jul 14 18:11:22.756316 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:11:22.756297    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286668344c09cd3eb4d034b7aed61498d3111a94e7cb79842a3f9bbcfa6efe1d\": container with ID starting with 286668344c09cd3eb4d034b7aed61498d3111a94e7cb79842a3f9bbcfa6efe1d not found: ID does not exist" containerID="286668344c09cd3eb4d034b7aed61498d3111a94e7cb79842a3f9bbcfa6efe1d"
Jul 14 18:11:22.756369 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.756326    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286668344c09cd3eb4d034b7aed61498d3111a94e7cb79842a3f9bbcfa6efe1d"} err="failed to get container status \"286668344c09cd3eb4d034b7aed61498d3111a94e7cb79842a3f9bbcfa6efe1d\": rpc error: code = NotFound desc = could not find container \"286668344c09cd3eb4d034b7aed61498d3111a94e7cb79842a3f9bbcfa6efe1d\": container with ID starting with 286668344c09cd3eb4d034b7aed61498d3111a94e7cb79842a3f9bbcfa6efe1d not found: ID does not exist"
Jul 14 18:11:22.775244 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.775221    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"]
Jul 14 18:11:22.782429 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.782408    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5a43f9-5947fd4557-g82ls"]
Jul 14 18:11:22.798873 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.798856    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc4g6\" (UniqueName: \"kubernetes.io/projected/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-kube-api-access-mc4g6\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:11:22.798938 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.798895    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-maas-api-tls\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:11:22.798938 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:22.798905    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c-openshift-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:11:23.784734 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:23.784701    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c" path="/var/lib/kubelet/pods/bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c/volumes"
Jul 14 18:11:24.286384 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:24.286343    2578 scope.go:117] "RemoveContainer" containerID="9133b28c12254d421cdeae0a49441f20c04fde5cb7e3eb229b9c088ffdd0beb8"
Jul 14 18:11:26.008529 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:26.008496    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:11:27.113559 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:27.113525    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:33.672497 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:33.672462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:33.774751 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:33.774723    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:37.854952 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:37.854915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:38.085234 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.085171    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:38.224559 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.224525    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n"]
Jul 14 18:11:38.224945 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.224932    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c" containerName="maas-api"
Jul 14 18:11:38.224997 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.224947    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c" containerName="maas-api"
Jul 14 18:11:38.224997 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.224960    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0dfb536-7f3a-4319-bbd3-ad40d63d7e30" containerName="revoke-keys"
Jul 14 18:11:38.224997 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.224965    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0dfb536-7f3a-4319-bbd3-ad40d63d7e30" containerName="revoke-keys"
Jul 14 18:11:38.225090 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.225027    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbcb60bb-ee0d-4990-9bc4-eee79c7f1b5c" containerName="maas-api"
Jul 14 18:11:38.225090 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.225034    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0dfb536-7f3a-4319-bbd3-ad40d63d7e30" containerName="revoke-keys"
Jul 14 18:11:38.228305 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.228289    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n"
Jul 14 18:11:38.231689 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.231672    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pd7w9\""
Jul 14 18:11:38.241731 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.241709    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n"]
Jul 14 18:11:38.344888 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.344846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e86dc48-8aa8-4a69-b21f-d1601cc8d31b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n\" (UID: \"2e86dc48-8aa8-4a69-b21f-d1601cc8d31b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n"
Jul 14 18:11:38.445994 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.445958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e86dc48-8aa8-4a69-b21f-d1601cc8d31b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n\" (UID: \"2e86dc48-8aa8-4a69-b21f-d1601cc8d31b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n"
Jul 14 18:11:38.446620 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.446599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e86dc48-8aa8-4a69-b21f-d1601cc8d31b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n\" (UID: \"2e86dc48-8aa8-4a69-b21f-d1601cc8d31b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n"
Jul 14 18:11:38.539159 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.539067    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n"
Jul 14 18:11:38.676412 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.676379    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n"]
Jul 14 18:11:38.676974 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:11:38.676949    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e86dc48_8aa8_4a69_b21f_d1601cc8d31b.slice/crio-74031c965a10ab6cebc5ff4afa91d9644cfaa955e4103d25a8d80b8836a8c369 WatchSource:0}: Error finding container 74031c965a10ab6cebc5ff4afa91d9644cfaa955e4103d25a8d80b8836a8c369: Status 404 returned error can't find the container with id 74031c965a10ab6cebc5ff4afa91d9644cfaa955e4103d25a8d80b8836a8c369
Jul 14 18:11:38.851868 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:38.851781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n" event={"ID":"2e86dc48-8aa8-4a69-b21f-d1601cc8d31b","Type":"ContainerStarted","Data":"74031c965a10ab6cebc5ff4afa91d9644cfaa955e4103d25a8d80b8836a8c369"}
Jul 14 18:11:39.858351 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:39.858318    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e86dc48-8aa8-4a69-b21f-d1601cc8d31b" containerID="36b9125f48c6ec7960fe5a57bec45b326b001f18743356f1a8a8207955484b50" exitCode=0
Jul 14 18:11:39.858813 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:39.858410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n" event={"ID":"2e86dc48-8aa8-4a69-b21f-d1601cc8d31b","Type":"ContainerDied","Data":"36b9125f48c6ec7960fe5a57bec45b326b001f18743356f1a8a8207955484b50"}
Jul 14 18:11:41.001099 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:41.001072    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n"
Jul 14 18:11:41.069521 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:41.069491    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e86dc48-8aa8-4a69-b21f-d1601cc8d31b-maas-api-service-ca\") pod \"2e86dc48-8aa8-4a69-b21f-d1601cc8d31b\" (UID: \"2e86dc48-8aa8-4a69-b21f-d1601cc8d31b\") "
Jul 14 18:11:41.069841 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:41.069819    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e86dc48-8aa8-4a69-b21f-d1601cc8d31b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e86dc48-8aa8-4a69-b21f-d1601cc8d31b" (UID: "2e86dc48-8aa8-4a69-b21f-d1601cc8d31b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:11:41.170602 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:41.170573    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e86dc48-8aa8-4a69-b21f-d1601cc8d31b-maas-api-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:11:41.871770 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:41.871745    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n"
Jul 14 18:11:41.871930 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:41.871735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n" event={"ID":"2e86dc48-8aa8-4a69-b21f-d1601cc8d31b","Type":"ContainerDied","Data":"74031c965a10ab6cebc5ff4afa91d9644cfaa955e4103d25a8d80b8836a8c369"}
Jul 14 18:11:41.871930 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:41.871869    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74031c965a10ab6cebc5ff4afa91d9644cfaa955e4103d25a8d80b8836a8c369"
Jul 14 18:11:48.795947 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:48.795890    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"]
Jul 14 18:11:48.796466 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:48.796238    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n" podUID="cc58c63d-266a-4e5f-a51c-01a8d10a32bf" containerName="maas-api" containerID="cri-o://b9fc1f2ff0ddee253afe5399f55353dc1de187da985a98a33c19df765f77d3b2" gracePeriod=30
Jul 14 18:11:48.834969 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:48.834938    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:48.913505 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:48.913334    2578 generic.go:356] "Generic (PLEG): container finished" podID="cc58c63d-266a-4e5f-a51c-01a8d10a32bf" containerID="b9fc1f2ff0ddee253afe5399f55353dc1de187da985a98a33c19df765f77d3b2" exitCode=0
Jul 14 18:11:48.913505 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:48.913425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n" event={"ID":"cc58c63d-266a-4e5f-a51c-01a8d10a32bf","Type":"ContainerDied","Data":"b9fc1f2ff0ddee253afe5399f55353dc1de187da985a98a33c19df765f77d3b2"}
Jul 14 18:11:49.050043 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:49.049979    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:11:49.144298 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:49.144263    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssplb\" (UniqueName: \"kubernetes.io/projected/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-kube-api-access-ssplb\") pod \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\" (UID: \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\") "
Jul 14 18:11:49.144479 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:49.144362    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-openshift-service-ca\") pod \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\" (UID: \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\") "
Jul 14 18:11:49.144479 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:49.144386    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-maas-api-tls\") pod \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\" (UID: \"cc58c63d-266a-4e5f-a51c-01a8d10a32bf\") "
Jul 14 18:11:49.144707 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:49.144683    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc58c63d-266a-4e5f-a51c-01a8d10a32bf" (UID: "cc58c63d-266a-4e5f-a51c-01a8d10a32bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:11:49.146630 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:49.146602    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc58c63d-266a-4e5f-a51c-01a8d10a32bf" (UID: "cc58c63d-266a-4e5f-a51c-01a8d10a32bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:11:49.146733 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:49.146625    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-kube-api-access-ssplb" (OuterVolumeSpecName: "kube-api-access-ssplb") pod "cc58c63d-266a-4e5f-a51c-01a8d10a32bf" (UID: "cc58c63d-266a-4e5f-a51c-01a8d10a32bf"). InnerVolumeSpecName "kube-api-access-ssplb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:11:49.244988 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:49.244957    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssplb\" (UniqueName: \"kubernetes.io/projected/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-kube-api-access-ssplb\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:11:49.244988 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:49.244984    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-openshift-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:11:49.244988 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:49.244994    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc58c63d-266a-4e5f-a51c-01a8d10a32bf-maas-api-tls\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:11:49.920297 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:49.920269    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"
Jul 14 18:11:49.920770 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:49.920301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n" event={"ID":"cc58c63d-266a-4e5f-a51c-01a8d10a32bf","Type":"ContainerDied","Data":"3cba9e8d0e759dc57d4358474cb49a40826e937fcba5ce32fd5139fedcf91695"}
Jul 14 18:11:49.920770 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:49.920367    2578 scope.go:117] "RemoveContainer" containerID="b9fc1f2ff0ddee253afe5399f55353dc1de187da985a98a33c19df765f77d3b2"
Jul 14 18:11:49.941685 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:49.941657    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"]
Jul 14 18:11:49.947198 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:49.947171    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9d840-6f6577f6b-v985n"]
Jul 14 18:11:51.620172 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:51.620131    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:11:51.783009 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:51.782977    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc58c63d-266a-4e5f-a51c-01a8d10a32bf" path="/var/lib/kubelet/pods/cc58c63d-266a-4e5f-a51c-01a8d10a32bf/volumes"
Jul 14 18:11:53.328610 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:53.328578    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:54.008969 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:54.008935    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:11:54.109282 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:11:54.109238    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:01.285884 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:01.285835    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:01.384709 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:01.384674    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:04.063841 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:04.063806    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:04.279414 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:04.279378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:04.631676 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:04.631647    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:05.075813 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:05.075763    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:05.207020 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:05.206986    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:07.776715 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.776674    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:07.874536 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.874495    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:07.886015 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.885986    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"]
Jul 14 18:12:07.886375 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.886363    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e86dc48-8aa8-4a69-b21f-d1601cc8d31b" containerName="revoke-keys"
Jul 14 18:12:07.886435 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.886377    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e86dc48-8aa8-4a69-b21f-d1601cc8d31b" containerName="revoke-keys"
Jul 14 18:12:07.886435 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.886401    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc58c63d-266a-4e5f-a51c-01a8d10a32bf" containerName="maas-api"
Jul 14 18:12:07.886435 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.886406    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc58c63d-266a-4e5f-a51c-01a8d10a32bf" containerName="maas-api"
Jul 14 18:12:07.886532 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.886465    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e86dc48-8aa8-4a69-b21f-d1601cc8d31b" containerName="revoke-keys"
Jul 14 18:12:07.886532 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.886475    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc58c63d-266a-4e5f-a51c-01a8d10a32bf" containerName="maas-api"
Jul 14 18:12:07.889723 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.889703    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:12:07.892950 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.892930    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-aa324e-serving-cert\""
Jul 14 18:12:07.892950 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.892942    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 18:12:07.893806 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.893781    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 18:12:07.893931 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.893826    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xxs8g\""
Jul 14 18:12:07.903262 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.903236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0479eb9-0695-49d3-8777-5e8803c65cac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl\" (UID: \"c0479eb9-0695-49d3-8777-5e8803c65cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:12:07.903399 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.903377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0479eb9-0695-49d3-8777-5e8803c65cac-maas-api-tls\") pod \"maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl\" (UID: \"c0479eb9-0695-49d3-8777-5e8803c65cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:12:07.903464 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.903429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vdbn\" (UniqueName: \"kubernetes.io/projected/c0479eb9-0695-49d3-8777-5e8803c65cac-kube-api-access-5vdbn\") pod \"maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl\" (UID: \"c0479eb9-0695-49d3-8777-5e8803c65cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:12:07.904156 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.904137    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"]
Jul 14 18:12:07.984579 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:07.984546    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:08.004801 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:08.004764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vdbn\" (UniqueName: \"kubernetes.io/projected/c0479eb9-0695-49d3-8777-5e8803c65cac-kube-api-access-5vdbn\") pod \"maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl\" (UID: \"c0479eb9-0695-49d3-8777-5e8803c65cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:12:08.005068 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:08.004816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0479eb9-0695-49d3-8777-5e8803c65cac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl\" (UID: \"c0479eb9-0695-49d3-8777-5e8803c65cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:12:08.005068 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:08.004894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0479eb9-0695-49d3-8777-5e8803c65cac-maas-api-tls\") pod \"maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl\" (UID: \"c0479eb9-0695-49d3-8777-5e8803c65cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:12:08.005068 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:12:08.004980    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-serving-cert: secret "maas-api-e2e-shared-a-aa324e-serving-cert" not found
Jul 14 18:12:08.005068 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:12:08.005044    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0479eb9-0695-49d3-8777-5e8803c65cac-maas-api-tls podName:c0479eb9-0695-49d3-8777-5e8803c65cac nodeName:}" failed. No retries permitted until 2026-07-14 18:12:08.505027077 +0000 UTC m=+2385.221420746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0479eb9-0695-49d3-8777-5e8803c65cac-maas-api-tls") pod "maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl" (UID: "c0479eb9-0695-49d3-8777-5e8803c65cac") : secret "maas-api-e2e-shared-a-aa324e-serving-cert" not found
Jul 14 18:12:08.005573 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:08.005555    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0479eb9-0695-49d3-8777-5e8803c65cac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl\" (UID: \"c0479eb9-0695-49d3-8777-5e8803c65cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:12:08.021783 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:08.021747    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vdbn\" (UniqueName: \"kubernetes.io/projected/c0479eb9-0695-49d3-8777-5e8803c65cac-kube-api-access-5vdbn\") pod \"maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl\" (UID: \"c0479eb9-0695-49d3-8777-5e8803c65cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:12:08.361250 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:08.361174    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:08.509280 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:08.509244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0479eb9-0695-49d3-8777-5e8803c65cac-maas-api-tls\") pod \"maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl\" (UID: \"c0479eb9-0695-49d3-8777-5e8803c65cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:12:08.511842 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:08.511816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0479eb9-0695-49d3-8777-5e8803c65cac-maas-api-tls\") pod \"maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl\" (UID: \"c0479eb9-0695-49d3-8777-5e8803c65cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:12:08.644241 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:08.644135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:08.747106 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:08.747075    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:08.799607 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:08.799566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:12:08.943908 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:08.943877    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"]
Jul 14 18:12:08.944057 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:12:08.943922    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0479eb9_0695_49d3_8777_5e8803c65cac.slice/crio-a8d1a2cec3a9d9b228acaae0d14296810b6285fd5cf7d73a300460e129a22398 WatchSource:0}: Error finding container a8d1a2cec3a9d9b228acaae0d14296810b6285fd5cf7d73a300460e129a22398: Status 404 returned error can't find the container with id a8d1a2cec3a9d9b228acaae0d14296810b6285fd5cf7d73a300460e129a22398
Jul 14 18:12:09.036916 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:09.036876    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl" event={"ID":"c0479eb9-0695-49d3-8777-5e8803c65cac","Type":"ContainerStarted","Data":"a8d1a2cec3a9d9b228acaae0d14296810b6285fd5cf7d73a300460e129a22398"}
Jul 14 18:12:10.044270 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:10.044234    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl" event={"ID":"c0479eb9-0695-49d3-8777-5e8803c65cac","Type":"ContainerStarted","Data":"26cb84000d171fae6c11c73ec3601732e8b5fbe1b881c229d68c40ee6a7a9104"}
Jul 14 18:12:10.044672 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:10.044292    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:12:10.075456 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:10.075401    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl" podStartSLOduration=2.646329981 podStartE2EDuration="3.075387559s" totalImagesPullingTime="429.057578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:12:10.072602526 +0000 UTC m=+2386.788996232" watchObservedRunningTime="2026-07-14 18:12:10.075387559 +0000 UTC m=+2386.791781251"
Jul 14 18:12:12.217476 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:12.217445    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:12.305568 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:12.305535    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:12:12.569273 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:12.569167    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:12.927480 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:12.927429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:13.237810 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:13.237725    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:15.943947 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:15.943907    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:15.952604 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:15.952579    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"]
Jul 14 18:12:15.956325 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:15.956309    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:12:15.960160 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:15.960142    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-877bd7-serving-cert\""
Jul 14 18:12:15.968009 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:15.967988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdacffa5-490a-4070-9ce9-44e4baa297d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw\" (UID: \"fdacffa5-490a-4070-9ce9-44e4baa297d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:12:15.968104 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:15.968014    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdacffa5-490a-4070-9ce9-44e4baa297d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw\" (UID: \"fdacffa5-490a-4070-9ce9-44e4baa297d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:12:15.968104 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:15.968080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9455\" (UniqueName: \"kubernetes.io/projected/fdacffa5-490a-4070-9ce9-44e4baa297d1-kube-api-access-k9455\") pod \"maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw\" (UID: \"fdacffa5-490a-4070-9ce9-44e4baa297d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:12:15.983899 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:15.983868    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"]
Jul 14 18:12:16.056778 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:16.056751    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:12:16.069129 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:16.069102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdacffa5-490a-4070-9ce9-44e4baa297d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw\" (UID: \"fdacffa5-490a-4070-9ce9-44e4baa297d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:12:16.069129 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:16.069136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdacffa5-490a-4070-9ce9-44e4baa297d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw\" (UID: \"fdacffa5-490a-4070-9ce9-44e4baa297d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:12:16.069397 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:16.069193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9455\" (UniqueName: \"kubernetes.io/projected/fdacffa5-490a-4070-9ce9-44e4baa297d1-kube-api-access-k9455\") pod \"maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw\" (UID: \"fdacffa5-490a-4070-9ce9-44e4baa297d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:12:16.069397 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:12:16.069262    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-serving-cert: secret "maas-api-e2e-shared-b-877bd7-serving-cert" not found
Jul 14 18:12:16.069397 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:12:16.069315    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdacffa5-490a-4070-9ce9-44e4baa297d1-maas-api-tls podName:fdacffa5-490a-4070-9ce9-44e4baa297d1 nodeName:}" failed. No retries permitted until 2026-07-14 18:12:16.569300448 +0000 UTC m=+2393.285694120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fdacffa5-490a-4070-9ce9-44e4baa297d1-maas-api-tls") pod "maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw" (UID: "fdacffa5-490a-4070-9ce9-44e4baa297d1") : secret "maas-api-e2e-shared-b-877bd7-serving-cert" not found
Jul 14 18:12:16.069828 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:16.069808    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdacffa5-490a-4070-9ce9-44e4baa297d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw\" (UID: \"fdacffa5-490a-4070-9ce9-44e4baa297d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:12:16.096614 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:16.096581    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9455\" (UniqueName: \"kubernetes.io/projected/fdacffa5-490a-4070-9ce9-44e4baa297d1-kube-api-access-k9455\") pod \"maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw\" (UID: \"fdacffa5-490a-4070-9ce9-44e4baa297d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:12:16.137867 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:16.137829    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:16.403057 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:16.402975    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:16.572676 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:16.572636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdacffa5-490a-4070-9ce9-44e4baa297d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw\" (UID: \"fdacffa5-490a-4070-9ce9-44e4baa297d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:12:16.575236 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:16.575183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdacffa5-490a-4070-9ce9-44e4baa297d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw\" (UID: \"fdacffa5-490a-4070-9ce9-44e4baa297d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:12:16.820901 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:16.820864    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:16.866913 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:16.866860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:12:17.024684 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:17.024656    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"]
Jul 14 18:12:17.025780 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:12:17.025751    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdacffa5_490a_4070_9ce9_44e4baa297d1.slice/crio-09a51811e3a9047c34c226ead609170918999a37d682bcb3654c8ae7334a04db WatchSource:0}: Error finding container 09a51811e3a9047c34c226ead609170918999a37d682bcb3654c8ae7334a04db: Status 404 returned error can't find the container with id 09a51811e3a9047c34c226ead609170918999a37d682bcb3654c8ae7334a04db
Jul 14 18:12:17.086607 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:17.086526    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw" event={"ID":"fdacffa5-490a-4070-9ce9-44e4baa297d1","Type":"ContainerStarted","Data":"09a51811e3a9047c34c226ead609170918999a37d682bcb3654c8ae7334a04db"}
Jul 14 18:12:17.127335 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:17.127304    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:18.095000 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:18.094959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw" event={"ID":"fdacffa5-490a-4070-9ce9-44e4baa297d1","Type":"ContainerStarted","Data":"928428acf2cd068770f51c851d58e87670f10d9dd2649fb08b56634c2ef28473"}
Jul 14 18:12:18.095409 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:18.095067    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:12:18.132521 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:18.132461    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw" podStartSLOduration=2.774072661 podStartE2EDuration="3.132444997s" totalImagesPullingTime="358.372336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:12:18.128995773 +0000 UTC m=+2394.845389478" watchObservedRunningTime="2026-07-14 18:12:18.132444997 +0000 UTC m=+2394.848838697"
Jul 14 18:12:20.333142 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:20.333109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:23.499692 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:23.499654    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:12:24.108698 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:24.108669    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:12:24.379353 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:24.379166    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 18:12:24.400904 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:24.383787    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 18:12:29.756894 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:29.756857    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:29.854667 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:29.854635    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:36.239434 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.239396    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"]
Jul 14 18:12:36.243002 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.242985    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.245542 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.245518    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-aa324e\"/\"kube-root-ca.crt\""
Jul 14 18:12:36.246524 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.246504    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-aa324e\"/\"default-dockercfg-lspvm\""
Jul 14 18:12:36.246609 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.246549    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-aa324e\"/\"openshift-service-ca.crt\""
Jul 14 18:12:36.246609 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.246555    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-aa324e\"/\"auth-test-model-aa324e-kserve-self-signed-certs\""
Jul 14 18:12:36.254958 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.254935    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"]
Jul 14 18:12:36.331346 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.331312    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:36.346526 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.346498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-model-cache\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.346680 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.346560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-tmp-dir\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.346680 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.346629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6662e7ad-6d1b-44af-b111-b77501b103f0-tls-certs\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.346680 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.346666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhxbm\" (UniqueName: \"kubernetes.io/projected/6662e7ad-6d1b-44af-b111-b77501b103f0-kube-api-access-dhxbm\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.346789 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.346697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-dshm\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.346789 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.346744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-home\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.447504 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.447473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-model-cache\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.447695 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.447540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-tmp-dir\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.447695 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.447565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6662e7ad-6d1b-44af-b111-b77501b103f0-tls-certs\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.447695 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.447587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhxbm\" (UniqueName: \"kubernetes.io/projected/6662e7ad-6d1b-44af-b111-b77501b103f0-kube-api-access-dhxbm\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.447695 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.447610    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-dshm\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.447695 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.447642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-home\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.447983 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.447963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-model-cache\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.447983 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.447975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-tmp-dir\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.448094 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.448021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-home\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.450113 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.450091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-dshm\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.450353 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.450336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6662e7ad-6d1b-44af-b111-b77501b103f0-tls-certs\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.456723 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.456699    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhxbm\" (UniqueName: \"kubernetes.io/projected/6662e7ad-6d1b-44af-b111-b77501b103f0-kube-api-access-dhxbm\") pod \"auth-test-model-aa324e-kserve-55fcb464dc-mk6fh\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") " pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.553420 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.553332    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:36.709708 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:36.709676    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"]
Jul 14 18:12:36.710944 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:12:36.710915    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6662e7ad_6d1b_44af_b111_b77501b103f0.slice/crio-0dc9844e6253be4e69c501610bafe523443dca20def9f35e0417cfb8c995e8c2 WatchSource:0}: Error finding container 0dc9844e6253be4e69c501610bafe523443dca20def9f35e0417cfb8c995e8c2: Status 404 returned error can't find the container with id 0dc9844e6253be4e69c501610bafe523443dca20def9f35e0417cfb8c995e8c2
Jul 14 18:12:37.210268 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:37.210227    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh" event={"ID":"6662e7ad-6d1b-44af-b111-b77501b103f0","Type":"ContainerStarted","Data":"5bec4cb60d07960d0511c17a5cfca264461125abc26d6a46b1810739ae133738"}
Jul 14 18:12:37.210443 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:37.210278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh" event={"ID":"6662e7ad-6d1b-44af-b111-b77501b103f0","Type":"ContainerStarted","Data":"0dc9844e6253be4e69c501610bafe523443dca20def9f35e0417cfb8c995e8c2"}
Jul 14 18:12:37.232058 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:37.232007    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh" podStartSLOduration=1.23199338 podStartE2EDuration="1.23199338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:12:37.230671241 +0000 UTC m=+2413.947064934" watchObservedRunningTime="2026-07-14 18:12:37.23199338 +0000 UTC m=+2413.948387071"
Jul 14 18:12:37.865678 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:37.865643    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:38.010003 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:38.009954    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:39.505951 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:39.505920    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:12:46.554408 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:46.554318    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:46.554408 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:46.554381    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:46.567285 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:46.567261    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:47.290437 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:47.290406    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:12:54.359272 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.359236    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"]
Jul 14 18:12:54.363966 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.363947    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.366630 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.366598    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-877bd7\"/\"openshift-service-ca.crt\""
Jul 14 18:12:54.367473 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.367453    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-877bd7\"/\"default-dockercfg-vxxnv\""
Jul 14 18:12:54.367595 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.367480    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-877bd7\"/\"auth-test-model-877bd7-kserve-self-signed-certs\""
Jul 14 18:12:54.367595 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.367523    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-877bd7\"/\"kube-root-ca.crt\""
Jul 14 18:12:54.379424 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.379399    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"]
Jul 14 18:12:54.495692 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.495654    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:12:54.524925 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.524878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vlh6\" (UniqueName: \"kubernetes.io/projected/c1c928af-4e96-4a92-b4cd-8416b656fbe6-kube-api-access-4vlh6\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.524925 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.524932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-dshm\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.525175 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.524996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-model-cache\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.525175 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.525035    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-home\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.525175 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.525161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-tmp-dir\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.525374 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.525250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c928af-4e96-4a92-b4cd-8416b656fbe6-tls-certs\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.626571 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.626473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-tmp-dir\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.626571 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.626518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c928af-4e96-4a92-b4cd-8416b656fbe6-tls-certs\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.626571 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.626563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vlh6\" (UniqueName: \"kubernetes.io/projected/c1c928af-4e96-4a92-b4cd-8416b656fbe6-kube-api-access-4vlh6\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.626858 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.626584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-dshm\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.626858 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.626599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-model-cache\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.626858 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.626630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-home\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.627026 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.626993    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-tmp-dir\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.627099 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.627077    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-model-cache\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.627160 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.627103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-home\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.629100 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.629074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-dshm\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.629429 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.629413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c928af-4e96-4a92-b4cd-8416b656fbe6-tls-certs\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.636283 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.636259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vlh6\" (UniqueName: \"kubernetes.io/projected/c1c928af-4e96-4a92-b4cd-8416b656fbe6-kube-api-access-4vlh6\") pod \"auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") " pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.674351 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.674315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:12:54.837976 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:54.837946    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"]
Jul 14 18:12:54.838803 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:12:54.838777    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1c928af_4e96_4a92_b4cd_8416b656fbe6.slice/crio-7be4df099feec2ed52548a2364cb49f88870dd335d0b497b3f8b5659a62d04b1 WatchSource:0}: Error finding container 7be4df099feec2ed52548a2364cb49f88870dd335d0b497b3f8b5659a62d04b1: Status 404 returned error can't find the container with id 7be4df099feec2ed52548a2364cb49f88870dd335d0b497b3f8b5659a62d04b1
Jul 14 18:12:55.327780 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:55.327742    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4" event={"ID":"c1c928af-4e96-4a92-b4cd-8416b656fbe6","Type":"ContainerStarted","Data":"b4ba09316633fe67faf20dc048e21a082af29cf99f671cb853327c2a44a3a3bf"}
Jul 14 18:12:55.327780 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:55.327781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4" event={"ID":"c1c928af-4e96-4a92-b4cd-8416b656fbe6","Type":"ContainerStarted","Data":"7be4df099feec2ed52548a2364cb49f88870dd335d0b497b3f8b5659a62d04b1"}
Jul 14 18:12:55.377276 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:55.377225    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4" podStartSLOduration=1.377188424 podStartE2EDuration="1.377188424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:12:55.368362046 +0000 UTC m=+2432.084755733" watchObservedRunningTime="2026-07-14 18:12:55.377188424 +0000 UTC m=+2432.093582115"
Jul 14 18:12:57.516331 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:12:57.516297    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:13:04.675358 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:13:04.675318    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:13:04.675873 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:13:04.675373    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:13:04.688368 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:13:04.688342    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:13:05.399418 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:13:05.399389    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:13:17.130271 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:13:17.130222    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:13:27.353150 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:13:27.353120    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:13:35.423604 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:13:35.423569    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:13:49.255268 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:13:49.255232    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:14:00.060444 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:14:00.060398    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:14:08.308132 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:14:08.308097    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:14:22.403749 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:14:22.403668    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:14:40.609114 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:14:40.609082    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:14:53.392586 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:14:53.392550    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"]
Jul 14 18:14:53.410167 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:14:53.410136    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:14:54.436976 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:14:54.436942    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"]
Jul 14 18:14:54.547689 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:14:54.547637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:14:56.520831 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:14:56.520791    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:15:08.413627 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.413577    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh" podUID="6662e7ad-6d1b-44af-b111-b77501b103f0" containerName="main" containerID="cri-o://5bec4cb60d07960d0511c17a5cfca264461125abc26d6a46b1810739ae133738" gracePeriod=45
Jul 14 18:15:08.678730 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.678702    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:15:08.741119 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.741092    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-home\") pod \"6662e7ad-6d1b-44af-b111-b77501b103f0\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") "
Jul 14 18:15:08.741283 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.741132    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-tmp-dir\") pod \"6662e7ad-6d1b-44af-b111-b77501b103f0\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") "
Jul 14 18:15:08.741283 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.741197    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-dshm\") pod \"6662e7ad-6d1b-44af-b111-b77501b103f0\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") "
Jul 14 18:15:08.741283 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.741272    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6662e7ad-6d1b-44af-b111-b77501b103f0-tls-certs\") pod \"6662e7ad-6d1b-44af-b111-b77501b103f0\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") "
Jul 14 18:15:08.741433 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.741293    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-model-cache\") pod \"6662e7ad-6d1b-44af-b111-b77501b103f0\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") "
Jul 14 18:15:08.741433 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.741318    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhxbm\" (UniqueName: \"kubernetes.io/projected/6662e7ad-6d1b-44af-b111-b77501b103f0-kube-api-access-dhxbm\") pod \"6662e7ad-6d1b-44af-b111-b77501b103f0\" (UID: \"6662e7ad-6d1b-44af-b111-b77501b103f0\") "
Jul 14 18:15:08.741433 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.741395    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-home" (OuterVolumeSpecName: "home") pod "6662e7ad-6d1b-44af-b111-b77501b103f0" (UID: "6662e7ad-6d1b-44af-b111-b77501b103f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:15:08.741539 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.741437    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6662e7ad-6d1b-44af-b111-b77501b103f0" (UID: "6662e7ad-6d1b-44af-b111-b77501b103f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:15:08.741600 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.741576    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "6662e7ad-6d1b-44af-b111-b77501b103f0" (UID: "6662e7ad-6d1b-44af-b111-b77501b103f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:15:08.741658 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.741644    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:15:08.741714 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.741661    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:15:08.741714 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.741675    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:15:08.743663 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.743629    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-dshm" (OuterVolumeSpecName: "dshm") pod "6662e7ad-6d1b-44af-b111-b77501b103f0" (UID: "6662e7ad-6d1b-44af-b111-b77501b103f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:15:08.743787 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.743687    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6662e7ad-6d1b-44af-b111-b77501b103f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6662e7ad-6d1b-44af-b111-b77501b103f0" (UID: "6662e7ad-6d1b-44af-b111-b77501b103f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:15:08.743787 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.743704    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6662e7ad-6d1b-44af-b111-b77501b103f0-kube-api-access-dhxbm" (OuterVolumeSpecName: "kube-api-access-dhxbm") pod "6662e7ad-6d1b-44af-b111-b77501b103f0" (UID: "6662e7ad-6d1b-44af-b111-b77501b103f0"). InnerVolumeSpecName "kube-api-access-dhxbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:15:08.842647 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.842614    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6662e7ad-6d1b-44af-b111-b77501b103f0-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:15:08.842647 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.842644    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6662e7ad-6d1b-44af-b111-b77501b103f0-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:15:08.842893 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:08.842661    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhxbm\" (UniqueName: \"kubernetes.io/projected/6662e7ad-6d1b-44af-b111-b77501b103f0-kube-api-access-dhxbm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:15:09.119666 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.119582    2578 generic.go:356] "Generic (PLEG): container finished" podID="6662e7ad-6d1b-44af-b111-b77501b103f0" containerID="5bec4cb60d07960d0511c17a5cfca264461125abc26d6a46b1810739ae133738" exitCode=0
Jul 14 18:15:09.119666 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.119629    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"
Jul 14 18:15:09.119847 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.119657    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh" event={"ID":"6662e7ad-6d1b-44af-b111-b77501b103f0","Type":"ContainerDied","Data":"5bec4cb60d07960d0511c17a5cfca264461125abc26d6a46b1810739ae133738"}
Jul 14 18:15:09.119847 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.119702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh" event={"ID":"6662e7ad-6d1b-44af-b111-b77501b103f0","Type":"ContainerDied","Data":"0dc9844e6253be4e69c501610bafe523443dca20def9f35e0417cfb8c995e8c2"}
Jul 14 18:15:09.119847 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.119720    2578 scope.go:117] "RemoveContainer" containerID="5bec4cb60d07960d0511c17a5cfca264461125abc26d6a46b1810739ae133738"
Jul 14 18:15:09.129651 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.129636    2578 scope.go:117] "RemoveContainer" containerID="5bec4cb60d07960d0511c17a5cfca264461125abc26d6a46b1810739ae133738"
Jul 14 18:15:09.129916 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:15:09.129897    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bec4cb60d07960d0511c17a5cfca264461125abc26d6a46b1810739ae133738\": container with ID starting with 5bec4cb60d07960d0511c17a5cfca264461125abc26d6a46b1810739ae133738 not found: ID does not exist" containerID="5bec4cb60d07960d0511c17a5cfca264461125abc26d6a46b1810739ae133738"
Jul 14 18:15:09.129986 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.129927    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bec4cb60d07960d0511c17a5cfca264461125abc26d6a46b1810739ae133738"} err="failed to get container status \"5bec4cb60d07960d0511c17a5cfca264461125abc26d6a46b1810739ae133738\": rpc error: code = NotFound desc = could not find container \"5bec4cb60d07960d0511c17a5cfca264461125abc26d6a46b1810739ae133738\": container with ID starting with 5bec4cb60d07960d0511c17a5cfca264461125abc26d6a46b1810739ae133738 not found: ID does not exist"
Jul 14 18:15:09.145903 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.145877    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"]
Jul 14 18:15:09.148457 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.148436    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/auth-test-model-aa324e-kserve-55fcb464dc-mk6fh"]
Jul 14 18:15:09.459228 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.459176    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4" podUID="c1c928af-4e96-4a92-b4cd-8416b656fbe6" containerName="main" containerID="cri-o://b4ba09316633fe67faf20dc048e21a082af29cf99f671cb853327c2a44a3a3bf" gracePeriod=45
Jul 14 18:15:09.719224 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.719138    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:15:09.783459 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.783424    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6662e7ad-6d1b-44af-b111-b77501b103f0" path="/var/lib/kubelet/pods/6662e7ad-6d1b-44af-b111-b77501b103f0/volumes"
Jul 14 18:15:09.852272 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.852239    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-tmp-dir\") pod \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") "
Jul 14 18:15:09.852272 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.852275    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-dshm\") pod \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") "
Jul 14 18:15:09.852512 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.852296    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-model-cache\") pod \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") "
Jul 14 18:15:09.852512 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.852320    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c928af-4e96-4a92-b4cd-8416b656fbe6-tls-certs\") pod \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") "
Jul 14 18:15:09.852512 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.852361    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vlh6\" (UniqueName: \"kubernetes.io/projected/c1c928af-4e96-4a92-b4cd-8416b656fbe6-kube-api-access-4vlh6\") pod \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") "
Jul 14 18:15:09.852512 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.852437    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-home\") pod \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\" (UID: \"c1c928af-4e96-4a92-b4cd-8416b656fbe6\") "
Jul 14 18:15:09.852706 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.852555    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1c928af-4e96-4a92-b4cd-8416b656fbe6" (UID: "c1c928af-4e96-4a92-b4cd-8416b656fbe6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:15:09.852706 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.852568    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1c928af-4e96-4a92-b4cd-8416b656fbe6" (UID: "c1c928af-4e96-4a92-b4cd-8416b656fbe6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:15:09.852826 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.852774    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-home" (OuterVolumeSpecName: "home") pod "c1c928af-4e96-4a92-b4cd-8416b656fbe6" (UID: "c1c928af-4e96-4a92-b4cd-8416b656fbe6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:15:09.852826 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.852785    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:15:09.852927 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.852831    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:15:09.854761 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.854736    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c928af-4e96-4a92-b4cd-8416b656fbe6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1c928af-4e96-4a92-b4cd-8416b656fbe6" (UID: "c1c928af-4e96-4a92-b4cd-8416b656fbe6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:15:09.854876 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.854766    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-dshm" (OuterVolumeSpecName: "dshm") pod "c1c928af-4e96-4a92-b4cd-8416b656fbe6" (UID: "c1c928af-4e96-4a92-b4cd-8416b656fbe6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:15:09.854876 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.854844    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c928af-4e96-4a92-b4cd-8416b656fbe6-kube-api-access-4vlh6" (OuterVolumeSpecName: "kube-api-access-4vlh6") pod "c1c928af-4e96-4a92-b4cd-8416b656fbe6" (UID: "c1c928af-4e96-4a92-b4cd-8416b656fbe6"). InnerVolumeSpecName "kube-api-access-4vlh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:15:09.953444 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.953408    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vlh6\" (UniqueName: \"kubernetes.io/projected/c1c928af-4e96-4a92-b4cd-8416b656fbe6-kube-api-access-4vlh6\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:15:09.953444 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.953436    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:15:09.953444 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.953446    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c928af-4e96-4a92-b4cd-8416b656fbe6-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:15:09.953661 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:09.953454    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c928af-4e96-4a92-b4cd-8416b656fbe6-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:15:10.127415 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:10.127328    2578 generic.go:356] "Generic (PLEG): container finished" podID="c1c928af-4e96-4a92-b4cd-8416b656fbe6" containerID="b4ba09316633fe67faf20dc048e21a082af29cf99f671cb853327c2a44a3a3bf" exitCode=0
Jul 14 18:15:10.127415 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:10.127378    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"
Jul 14 18:15:10.127415 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:10.127396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4" event={"ID":"c1c928af-4e96-4a92-b4cd-8416b656fbe6","Type":"ContainerDied","Data":"b4ba09316633fe67faf20dc048e21a082af29cf99f671cb853327c2a44a3a3bf"}
Jul 14 18:15:10.127638 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:10.127432    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4" event={"ID":"c1c928af-4e96-4a92-b4cd-8416b656fbe6","Type":"ContainerDied","Data":"7be4df099feec2ed52548a2364cb49f88870dd335d0b497b3f8b5659a62d04b1"}
Jul 14 18:15:10.127638 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:10.127456    2578 scope.go:117] "RemoveContainer" containerID="b4ba09316633fe67faf20dc048e21a082af29cf99f671cb853327c2a44a3a3bf"
Jul 14 18:15:10.138264 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:10.138246    2578 scope.go:117] "RemoveContainer" containerID="b4ba09316633fe67faf20dc048e21a082af29cf99f671cb853327c2a44a3a3bf"
Jul 14 18:15:10.138539 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:15:10.138516    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4ba09316633fe67faf20dc048e21a082af29cf99f671cb853327c2a44a3a3bf\": container with ID starting with b4ba09316633fe67faf20dc048e21a082af29cf99f671cb853327c2a44a3a3bf not found: ID does not exist" containerID="b4ba09316633fe67faf20dc048e21a082af29cf99f671cb853327c2a44a3a3bf"
Jul 14 18:15:10.138625 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:10.138544    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4ba09316633fe67faf20dc048e21a082af29cf99f671cb853327c2a44a3a3bf"} err="failed to get container status \"b4ba09316633fe67faf20dc048e21a082af29cf99f671cb853327c2a44a3a3bf\": rpc error: code = NotFound desc = could not find container \"b4ba09316633fe67faf20dc048e21a082af29cf99f671cb853327c2a44a3a3bf\": container with ID starting with b4ba09316633fe67faf20dc048e21a082af29cf99f671cb853327c2a44a3a3bf not found: ID does not exist"
Jul 14 18:15:10.153473 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:10.153446    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"]
Jul 14 18:15:10.158416 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:10.158389    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/auth-test-model-877bd7-kserve-69f7b7d6c5-5rhm4"]
Jul 14 18:15:11.302103 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:11.302068    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:15:11.784067 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:11.784033    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1c928af-4e96-4a92-b4cd-8416b656fbe6" path="/var/lib/kubelet/pods/c1c928af-4e96-4a92-b4cd-8416b656fbe6/volumes"
Jul 14 18:15:15.845864 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:15.845831    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"]
Jul 14 18:15:15.846252 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:15.846238    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6662e7ad-6d1b-44af-b111-b77501b103f0" containerName="main"
Jul 14 18:15:15.846252 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:15.846250    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6662e7ad-6d1b-44af-b111-b77501b103f0" containerName="main"
Jul 14 18:15:15.846335 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:15.846275    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c928af-4e96-4a92-b4cd-8416b656fbe6" containerName="main"
Jul 14 18:15:15.846335 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:15.846280    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c928af-4e96-4a92-b4cd-8416b656fbe6" containerName="main"
Jul 14 18:15:15.846402 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:15.846339    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6662e7ad-6d1b-44af-b111-b77501b103f0" containerName="main"
Jul 14 18:15:15.846402 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:15.846350    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1c928af-4e96-4a92-b4cd-8416b656fbe6" containerName="main"
Jul 14 18:15:15.850772 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:15.850754    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:15.853761 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:15.853737    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-aa324e\"/\"kube-root-ca.crt\""
Jul 14 18:15:15.854502 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:15.854485    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-aa324e\"/\"openshift-service-ca.crt\""
Jul 14 18:15:15.854574 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:15.854558    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-aa324e\"/\"sub-test-model-aa324e-kserve-self-signed-certs\""
Jul 14 18:15:15.854625 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:15.854578    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-aa324e\"/\"default-dockercfg-lspvm\""
Jul 14 18:15:15.865247 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:15.865224    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"]
Jul 14 18:15:15.883116 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:15.883089    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:15:16.010713 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.010678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-tmp-dir\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.010901 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.010724    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-dshm\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.010901 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.010757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw8q6\" (UniqueName: \"kubernetes.io/projected/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-kube-api-access-qw8q6\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.010901 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.010825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-model-cache\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.010901 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.010867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-tls-certs\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.010901 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.010890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-home\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.112232 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.112123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-tmp-dir\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.112232 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.112169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-dshm\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.112232 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.112192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8q6\" (UniqueName: \"kubernetes.io/projected/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-kube-api-access-qw8q6\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.112482 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.112325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-model-cache\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.112482 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.112366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-tls-certs\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.112482 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.112395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-home\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.112663 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.112640    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-tmp-dir\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.112796 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.112775    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-model-cache\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.112852 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.112778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-home\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.114570 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.114550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-dshm\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.114923 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.114895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-tls-certs\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.126938 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.126902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw8q6\" (UniqueName: \"kubernetes.io/projected/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-kube-api-access-qw8q6\") pod \"sub-test-model-aa324e-kserve-d7564558c-lzxsx\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") " pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.159515 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.159487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:16.305735 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:16.305705    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"]
Jul 14 18:15:16.306699 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:15:16.306669    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bbbbc10_ccd5_4ebf_99ba_0283883fffee.slice/crio-a9a75f974df188ab1703758b2b9a4cd842e21fa4bd2ec55d3879bc9a09351181 WatchSource:0}: Error finding container a9a75f974df188ab1703758b2b9a4cd842e21fa4bd2ec55d3879bc9a09351181: Status 404 returned error can't find the container with id a9a75f974df188ab1703758b2b9a4cd842e21fa4bd2ec55d3879bc9a09351181
Jul 14 18:15:17.178731 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:17.178696    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx" event={"ID":"8bbbbc10-ccd5-4ebf-99ba-0283883fffee","Type":"ContainerStarted","Data":"7525c657ca61c906d5f949d5b5c624817fbc5ba5661dab9cc51929c2daa32d7b"}
Jul 14 18:15:17.179110 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:17.178744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx" event={"ID":"8bbbbc10-ccd5-4ebf-99ba-0283883fffee","Type":"ContainerStarted","Data":"a9a75f974df188ab1703758b2b9a4cd842e21fa4bd2ec55d3879bc9a09351181"}
Jul 14 18:15:17.220464 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:17.220413    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx" podStartSLOduration=2.220397515 podStartE2EDuration="2.220397515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:15:17.218124862 +0000 UTC m=+2573.934518554" watchObservedRunningTime="2026-07-14 18:15:17.220397515 +0000 UTC m=+2573.936791206"
Jul 14 18:15:20.640137 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:20.640102    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:15:26.159758 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:26.159714    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:26.160235 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:26.159807    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:26.172260 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:26.172233    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:26.244133 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:26.244102    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:15:28.595089 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:28.595048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:15:28.691063 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:28.691031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:15:38.052985 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.052948    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"]
Jul 14 18:15:38.056473 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.056449    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.059035 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.059014    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-877bd7\"/\"kube-root-ca.crt\""
Jul 14 18:15:38.060046 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.060019    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-877bd7\"/\"openshift-service-ca.crt\""
Jul 14 18:15:38.060046 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.060041    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-877bd7\"/\"sub-test-model-877bd7-kserve-self-signed-certs\""
Jul 14 18:15:38.060186 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.060053    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-877bd7\"/\"default-dockercfg-vxxnv\""
Jul 14 18:15:38.073755 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.073731    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"]
Jul 14 18:15:38.113836 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.113798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzqp\" (UniqueName: \"kubernetes.io/projected/ebeae839-07b7-4935-8f7c-21d1fefcb242-kube-api-access-6pzqp\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.114000 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.113849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebeae839-07b7-4935-8f7c-21d1fefcb242-tls-certs\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.114000 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.113901    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-dshm\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.114000 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.113964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-home\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.114000 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.113983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-tmp-dir\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.114139 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.114016    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-model-cache\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.136627 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.136595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:15:38.214999 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.214966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-home\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.214999 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.214996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-tmp-dir\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.215270 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.215247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-model-cache\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.215386 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.215369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzqp\" (UniqueName: \"kubernetes.io/projected/ebeae839-07b7-4935-8f7c-21d1fefcb242-kube-api-access-6pzqp\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.215433 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.215415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebeae839-07b7-4935-8f7c-21d1fefcb242-tls-certs\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.215480 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.215434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-home\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.215480 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.215447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-tmp-dir\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.215548 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.215492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-dshm\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.215548 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.215512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-model-cache\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.217766 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.217738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-dshm\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.218043 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.218024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebeae839-07b7-4935-8f7c-21d1fefcb242-tls-certs\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.226254 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.226229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzqp\" (UniqueName: \"kubernetes.io/projected/ebeae839-07b7-4935-8f7c-21d1fefcb242-kube-api-access-6pzqp\") pod \"sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") " pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.366159 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.366065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:38.506433 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:38.506401    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"]
Jul 14 18:15:38.507193 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:15:38.507169    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebeae839_07b7_4935_8f7c_21d1fefcb242.slice/crio-c07ac4675001c166f0dea25036fe28ed74032973e33d0c8f98f19fafd193cad6 WatchSource:0}: Error finding container c07ac4675001c166f0dea25036fe28ed74032973e33d0c8f98f19fafd193cad6: Status 404 returned error can't find the container with id c07ac4675001c166f0dea25036fe28ed74032973e33d0c8f98f19fafd193cad6
Jul 14 18:15:39.309904 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:39.309868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp" event={"ID":"ebeae839-07b7-4935-8f7c-21d1fefcb242","Type":"ContainerStarted","Data":"3178f27c1da28d30db62c66ead379789851aa94c6ed01e227bda7fd94eb96f95"}
Jul 14 18:15:39.310356 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:39.309914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp" event={"ID":"ebeae839-07b7-4935-8f7c-21d1fefcb242","Type":"ContainerStarted","Data":"c07ac4675001c166f0dea25036fe28ed74032973e33d0c8f98f19fafd193cad6"}
Jul 14 18:15:39.333440 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:39.333380    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp" podStartSLOduration=1.3333637839999999 podStartE2EDuration="1.333363784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:15:39.330996395 +0000 UTC m=+2596.047390086" watchObservedRunningTime="2026-07-14 18:15:39.333363784 +0000 UTC m=+2596.049757475"
Jul 14 18:15:42.006169 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:42.006135    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:15:48.366331 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:48.366249    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:48.366331 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:48.366287    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:48.378979 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:48.378952    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:15:49.381950 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:15:49.381924    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:16:00.912542 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:00.912508    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:16:19.897932 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:19.897895    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:16:24.278815 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:24.278778    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"]
Jul 14 18:16:24.307134 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:24.307069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:16:25.176416 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:25.176387    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"]
Jul 14 18:16:25.194040 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:25.193998    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:16:31.211275 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:31.211248    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:16:37.311723 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:37.311681    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:16:37.947965 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:37.947928    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:16:37.948276 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:37.948252    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"]
Jul 14 18:16:37.951970 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:37.951952    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:37.965769 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:37.965750    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-aa324e\"/\"rate-test-model-aa324e-kserve-self-signed-certs\""
Jul 14 18:16:37.988129 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:37.988095    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"]
Jul 14 18:16:38.061596 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.061554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-home\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.061596 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.061594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-model-cache\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.061841 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.061641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fgzk\" (UniqueName: \"kubernetes.io/projected/d3d0081d-6030-4bef-908a-269f6f90cd95-kube-api-access-9fgzk\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.061841 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.061669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-tmp-dir\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.061841 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.061762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d0081d-6030-4bef-908a-269f6f90cd95-tls-certs\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.061841 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.061808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-dshm\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.162613 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.162580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-home\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.162613 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.162613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-model-cache\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.162821 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.162633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fgzk\" (UniqueName: \"kubernetes.io/projected/d3d0081d-6030-4bef-908a-269f6f90cd95-kube-api-access-9fgzk\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.162821 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.162768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-tmp-dir\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.162916 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.162851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d0081d-6030-4bef-908a-269f6f90cd95-tls-certs\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.162916 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.162897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-dshm\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.163023 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.162982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-home\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.163088 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.163067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-model-cache\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.163147 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.163129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-tmp-dir\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.165358 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.165334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-dshm\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.165555 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.165535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d0081d-6030-4bef-908a-269f6f90cd95-tls-certs\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.173262 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.173234    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fgzk\" (UniqueName: \"kubernetes.io/projected/d3d0081d-6030-4bef-908a-269f6f90cd95-kube-api-access-9fgzk\") pod \"rate-test-model-aa324e-kserve-788f6d97dc-m55vg\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") " pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.260993 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.260908    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:38.406479 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.406453    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"]
Jul 14 18:16:38.408838 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:16:38.408809    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3d0081d_6030_4bef_908a_269f6f90cd95.slice/crio-09b0aa3805c6bf620dda62620a4d372427fbf31b2889235cb4399964d50f176d WatchSource:0}: Error finding container 09b0aa3805c6bf620dda62620a4d372427fbf31b2889235cb4399964d50f176d: Status 404 returned error can't find the container with id 09b0aa3805c6bf620dda62620a4d372427fbf31b2889235cb4399964d50f176d
Jul 14 18:16:38.671153 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.671121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg" event={"ID":"d3d0081d-6030-4bef-908a-269f6f90cd95","Type":"ContainerStarted","Data":"d77bb1b693519fad62e1c079bc0c78560b93110dbc3f389cdb4d2bde945a65ca"}
Jul 14 18:16:38.671352 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.671160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg" event={"ID":"d3d0081d-6030-4bef-908a-269f6f90cd95","Type":"ContainerStarted","Data":"09b0aa3805c6bf620dda62620a4d372427fbf31b2889235cb4399964d50f176d"}
Jul 14 18:16:38.696409 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:38.696357    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg" podStartSLOduration=1.696341251 podStartE2EDuration="1.696341251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:16:38.693470733 +0000 UTC m=+2655.409864424" watchObservedRunningTime="2026-07-14 18:16:38.696341251 +0000 UTC m=+2655.412734943"
Jul 14 18:16:39.300852 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.300809    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx" podUID="8bbbbc10-ccd5-4ebf-99ba-0283883fffee" containerName="main" containerID="cri-o://7525c657ca61c906d5f949d5b5c624817fbc5ba5661dab9cc51929c2daa32d7b" gracePeriod=45
Jul 14 18:16:39.569003 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.568977    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:16:39.675791 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.675763    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-tmp-dir\") pod \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") "
Jul 14 18:16:39.675985 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.675851    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-home\") pod \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") "
Jul 14 18:16:39.675985 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.675877    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-model-cache\") pod \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") "
Jul 14 18:16:39.675985 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.675897    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw8q6\" (UniqueName: \"kubernetes.io/projected/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-kube-api-access-qw8q6\") pod \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") "
Jul 14 18:16:39.675985 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.675934    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-dshm\") pod \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") "
Jul 14 18:16:39.675985 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.675959    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-tls-certs\") pod \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\" (UID: \"8bbbbc10-ccd5-4ebf-99ba-0283883fffee\") "
Jul 14 18:16:39.676267 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.676032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8bbbbc10-ccd5-4ebf-99ba-0283883fffee" (UID: "8bbbbc10-ccd5-4ebf-99ba-0283883fffee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:16:39.676267 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.676191    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-model-cache" (OuterVolumeSpecName: "model-cache") pod "8bbbbc10-ccd5-4ebf-99ba-0283883fffee" (UID: "8bbbbc10-ccd5-4ebf-99ba-0283883fffee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:16:39.676382 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.676312    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:16:39.676382 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.676332    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:16:39.676487 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.676402    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-home" (OuterVolumeSpecName: "home") pod "8bbbbc10-ccd5-4ebf-99ba-0283883fffee" (UID: "8bbbbc10-ccd5-4ebf-99ba-0283883fffee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:16:39.678613 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.678578    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-kube-api-access-qw8q6" (OuterVolumeSpecName: "kube-api-access-qw8q6") pod "8bbbbc10-ccd5-4ebf-99ba-0283883fffee" (UID: "8bbbbc10-ccd5-4ebf-99ba-0283883fffee"). InnerVolumeSpecName "kube-api-access-qw8q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:16:39.678729 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.678659    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-dshm" (OuterVolumeSpecName: "dshm") pod "8bbbbc10-ccd5-4ebf-99ba-0283883fffee" (UID: "8bbbbc10-ccd5-4ebf-99ba-0283883fffee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:16:39.678729 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.678680    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8bbbbc10-ccd5-4ebf-99ba-0283883fffee" (UID: "8bbbbc10-ccd5-4ebf-99ba-0283883fffee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:16:39.680526 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.680502    2578 generic.go:356] "Generic (PLEG): container finished" podID="8bbbbc10-ccd5-4ebf-99ba-0283883fffee" containerID="7525c657ca61c906d5f949d5b5c624817fbc5ba5661dab9cc51929c2daa32d7b" exitCode=0
Jul 14 18:16:39.680606 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.680550    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"
Jul 14 18:16:39.680606 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.680577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx" event={"ID":"8bbbbc10-ccd5-4ebf-99ba-0283883fffee","Type":"ContainerDied","Data":"7525c657ca61c906d5f949d5b5c624817fbc5ba5661dab9cc51929c2daa32d7b"}
Jul 14 18:16:39.680678 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.680617    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx" event={"ID":"8bbbbc10-ccd5-4ebf-99ba-0283883fffee","Type":"ContainerDied","Data":"a9a75f974df188ab1703758b2b9a4cd842e21fa4bd2ec55d3879bc9a09351181"}
Jul 14 18:16:39.680678 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.680634    2578 scope.go:117] "RemoveContainer" containerID="7525c657ca61c906d5f949d5b5c624817fbc5ba5661dab9cc51929c2daa32d7b"
Jul 14 18:16:39.695392 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.695376    2578 scope.go:117] "RemoveContainer" containerID="7525c657ca61c906d5f949d5b5c624817fbc5ba5661dab9cc51929c2daa32d7b"
Jul 14 18:16:39.695770 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:16:39.695713    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7525c657ca61c906d5f949d5b5c624817fbc5ba5661dab9cc51929c2daa32d7b\": container with ID starting with 7525c657ca61c906d5f949d5b5c624817fbc5ba5661dab9cc51929c2daa32d7b not found: ID does not exist" containerID="7525c657ca61c906d5f949d5b5c624817fbc5ba5661dab9cc51929c2daa32d7b"
Jul 14 18:16:39.695770 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.695750    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7525c657ca61c906d5f949d5b5c624817fbc5ba5661dab9cc51929c2daa32d7b"} err="failed to get container status \"7525c657ca61c906d5f949d5b5c624817fbc5ba5661dab9cc51929c2daa32d7b\": rpc error: code = NotFound desc = could not find container \"7525c657ca61c906d5f949d5b5c624817fbc5ba5661dab9cc51929c2daa32d7b\": container with ID starting with 7525c657ca61c906d5f949d5b5c624817fbc5ba5661dab9cc51929c2daa32d7b not found: ID does not exist"
Jul 14 18:16:39.716817 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.716794    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"]
Jul 14 18:16:39.724401 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.724380    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/sub-test-model-aa324e-kserve-d7564558c-lzxsx"]
Jul 14 18:16:39.777737 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.777717    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:16:39.777866 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.777833    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw8q6\" (UniqueName: \"kubernetes.io/projected/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-kube-api-access-qw8q6\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:16:39.777931 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.777912    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:16:39.777931 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.777924    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bbbbc10-ccd5-4ebf-99ba-0283883fffee-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:16:39.783970 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:39.783946    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bbbbc10-ccd5-4ebf-99ba-0283883fffee" path="/var/lib/kubelet/pods/8bbbbc10-ccd5-4ebf-99ba-0283883fffee/volumes"
Jul 14 18:16:40.197511 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.197467    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp" podUID="ebeae839-07b7-4935-8f7c-21d1fefcb242" containerName="main" containerID="cri-o://3178f27c1da28d30db62c66ead379789851aa94c6ed01e227bda7fd94eb96f95" gracePeriod=45
Jul 14 18:16:40.453426 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.453365    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:16:40.585475 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.585446    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pzqp\" (UniqueName: \"kubernetes.io/projected/ebeae839-07b7-4935-8f7c-21d1fefcb242-kube-api-access-6pzqp\") pod \"ebeae839-07b7-4935-8f7c-21d1fefcb242\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") "
Jul 14 18:16:40.585904 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.585495    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-model-cache\") pod \"ebeae839-07b7-4935-8f7c-21d1fefcb242\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") "
Jul 14 18:16:40.585904 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.585551    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebeae839-07b7-4935-8f7c-21d1fefcb242-tls-certs\") pod \"ebeae839-07b7-4935-8f7c-21d1fefcb242\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") "
Jul 14 18:16:40.585904 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.585604    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-dshm\") pod \"ebeae839-07b7-4935-8f7c-21d1fefcb242\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") "
Jul 14 18:16:40.585904 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.585626    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-home\") pod \"ebeae839-07b7-4935-8f7c-21d1fefcb242\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") "
Jul 14 18:16:40.585904 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.585642    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-tmp-dir\") pod \"ebeae839-07b7-4935-8f7c-21d1fefcb242\" (UID: \"ebeae839-07b7-4935-8f7c-21d1fefcb242\") "
Jul 14 18:16:40.585904 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.585839    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-model-cache" (OuterVolumeSpecName: "model-cache") pod "ebeae839-07b7-4935-8f7c-21d1fefcb242" (UID: "ebeae839-07b7-4935-8f7c-21d1fefcb242"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:16:40.586252 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.585956    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ebeae839-07b7-4935-8f7c-21d1fefcb242" (UID: "ebeae839-07b7-4935-8f7c-21d1fefcb242"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:16:40.586252 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.585970    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-home" (OuterVolumeSpecName: "home") pod "ebeae839-07b7-4935-8f7c-21d1fefcb242" (UID: "ebeae839-07b7-4935-8f7c-21d1fefcb242"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:16:40.587735 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.587712    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-dshm" (OuterVolumeSpecName: "dshm") pod "ebeae839-07b7-4935-8f7c-21d1fefcb242" (UID: "ebeae839-07b7-4935-8f7c-21d1fefcb242"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:16:40.587735 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.587715    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebeae839-07b7-4935-8f7c-21d1fefcb242-kube-api-access-6pzqp" (OuterVolumeSpecName: "kube-api-access-6pzqp") pod "ebeae839-07b7-4935-8f7c-21d1fefcb242" (UID: "ebeae839-07b7-4935-8f7c-21d1fefcb242"). InnerVolumeSpecName "kube-api-access-6pzqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:16:40.587888 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.587796    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebeae839-07b7-4935-8f7c-21d1fefcb242-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebeae839-07b7-4935-8f7c-21d1fefcb242" (UID: "ebeae839-07b7-4935-8f7c-21d1fefcb242"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:16:40.687071 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.687032    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzqp\" (UniqueName: \"kubernetes.io/projected/ebeae839-07b7-4935-8f7c-21d1fefcb242-kube-api-access-6pzqp\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:16:40.687071 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.687060    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:16:40.687071 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.687074    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebeae839-07b7-4935-8f7c-21d1fefcb242-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:16:40.687071 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.687081    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:16:40.687398 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.687088    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:16:40.687398 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.687099    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebeae839-07b7-4935-8f7c-21d1fefcb242-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:16:40.689246 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.689189    2578 generic.go:356] "Generic (PLEG): container finished" podID="ebeae839-07b7-4935-8f7c-21d1fefcb242" containerID="3178f27c1da28d30db62c66ead379789851aa94c6ed01e227bda7fd94eb96f95" exitCode=0
Jul 14 18:16:40.689371 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.689242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp" event={"ID":"ebeae839-07b7-4935-8f7c-21d1fefcb242","Type":"ContainerDied","Data":"3178f27c1da28d30db62c66ead379789851aa94c6ed01e227bda7fd94eb96f95"}
Jul 14 18:16:40.689371 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.689263    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"
Jul 14 18:16:40.689371 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.689289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp" event={"ID":"ebeae839-07b7-4935-8f7c-21d1fefcb242","Type":"ContainerDied","Data":"c07ac4675001c166f0dea25036fe28ed74032973e33d0c8f98f19fafd193cad6"}
Jul 14 18:16:40.689371 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.689307    2578 scope.go:117] "RemoveContainer" containerID="3178f27c1da28d30db62c66ead379789851aa94c6ed01e227bda7fd94eb96f95"
Jul 14 18:16:40.698850 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.698825    2578 scope.go:117] "RemoveContainer" containerID="3178f27c1da28d30db62c66ead379789851aa94c6ed01e227bda7fd94eb96f95"
Jul 14 18:16:40.699098 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:16:40.699081    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3178f27c1da28d30db62c66ead379789851aa94c6ed01e227bda7fd94eb96f95\": container with ID starting with 3178f27c1da28d30db62c66ead379789851aa94c6ed01e227bda7fd94eb96f95 not found: ID does not exist" containerID="3178f27c1da28d30db62c66ead379789851aa94c6ed01e227bda7fd94eb96f95"
Jul 14 18:16:40.699164 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.699105    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3178f27c1da28d30db62c66ead379789851aa94c6ed01e227bda7fd94eb96f95"} err="failed to get container status \"3178f27c1da28d30db62c66ead379789851aa94c6ed01e227bda7fd94eb96f95\": rpc error: code = NotFound desc = could not find container \"3178f27c1da28d30db62c66ead379789851aa94c6ed01e227bda7fd94eb96f95\": container with ID starting with 3178f27c1da28d30db62c66ead379789851aa94c6ed01e227bda7fd94eb96f95 not found: ID does not exist"
Jul 14 18:16:40.714560 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.714508    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"]
Jul 14 18:16:40.724885 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:40.724862    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/sub-test-model-877bd7-kserve-6b69c7fdbf-w7znp"]
Jul 14 18:16:41.789167 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:41.785976    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebeae839-07b7-4935-8f7c-21d1fefcb242" path="/var/lib/kubelet/pods/ebeae839-07b7-4935-8f7c-21d1fefcb242/volumes"
Jul 14 18:16:48.262037 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:48.262004    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:48.262526 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:48.262069    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:48.275606 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:48.275583    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:48.749416 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:48.749390    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:16:49.528354 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:49.528317    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:16:53.461860 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:16:53.461825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:17:10.846197 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.846164    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"]
Jul 14 18:17:10.846795 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.846775    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bbbbc10-ccd5-4ebf-99ba-0283883fffee" containerName="main"
Jul 14 18:17:10.846889 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.846799    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bbbbc10-ccd5-4ebf-99ba-0283883fffee" containerName="main"
Jul 14 18:17:10.846889 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.846816    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebeae839-07b7-4935-8f7c-21d1fefcb242" containerName="main"
Jul 14 18:17:10.846889 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.846824    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebeae839-07b7-4935-8f7c-21d1fefcb242" containerName="main"
Jul 14 18:17:10.847048 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.846942    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bbbbc10-ccd5-4ebf-99ba-0283883fffee" containerName="main"
Jul 14 18:17:10.847048 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.846954    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebeae839-07b7-4935-8f7c-21d1fefcb242" containerName="main"
Jul 14 18:17:10.850471 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.850450    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:10.853109 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.853087    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-877bd7\"/\"kube-root-ca.crt\""
Jul 14 18:17:10.853905 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.853886    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-877bd7\"/\"default-dockercfg-vxxnv\""
Jul 14 18:17:10.853905 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.853899    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-877bd7\"/\"rate-test-model-877bd7-kserve-self-signed-certs\""
Jul 14 18:17:10.854062 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.853925    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-877bd7\"/\"openshift-service-ca.crt\""
Jul 14 18:17:10.860533 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.860512    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"]
Jul 14 18:17:10.969221 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.969161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-home\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:10.969221 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.969224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-tmp-dir\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:10.969486 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.969271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-model-cache\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:10.969486 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.969332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-tls-certs\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:10.969486 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.969382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr5tn\" (UniqueName: \"kubernetes.io/projected/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-kube-api-access-dr5tn\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:10.969486 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:10.969410    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-dshm\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:11.070614 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.070580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-home\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:11.070811 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.070629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-tmp-dir\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:11.070811 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.070673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-model-cache\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:11.070811 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.070732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-tls-certs\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:11.070811 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.070779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr5tn\" (UniqueName: \"kubernetes.io/projected/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-kube-api-access-dr5tn\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:11.071070 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.071047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-home\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:11.071126 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.071076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-tmp-dir\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:11.071126 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.071092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-model-cache\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:11.071126 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.070812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-dshm\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:11.073438 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.073416    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-dshm\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:11.073544 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.073480    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-tls-certs\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:11.080047 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.080026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr5tn\" (UniqueName: \"kubernetes.io/projected/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-kube-api-access-dr5tn\") pod \"rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") " pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:11.160513 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.160420    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:11.315910 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.315880    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"]
Jul 14 18:17:11.316129 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:17:11.316106    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddacd2e0_5ad5_46b1_9825_9f9df528d1cb.slice/crio-74658e091e0467124f2f1e31058812cfe7172b685f5093e0cc226746be06824c WatchSource:0}: Error finding container 74658e091e0467124f2f1e31058812cfe7172b685f5093e0cc226746be06824c: Status 404 returned error can't find the container with id 74658e091e0467124f2f1e31058812cfe7172b685f5093e0cc226746be06824c
Jul 14 18:17:11.881649 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.881611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr" event={"ID":"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb","Type":"ContainerStarted","Data":"4d831200227b4fae9d3f2754832955cd6b82fefbc55b393de3e602664e2117a4"}
Jul 14 18:17:11.882085 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.881672    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr" event={"ID":"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb","Type":"ContainerStarted","Data":"74658e091e0467124f2f1e31058812cfe7172b685f5093e0cc226746be06824c"}
Jul 14 18:17:11.908677 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:11.908620    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr" podStartSLOduration=1.908602427 podStartE2EDuration="1.908602427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:17:11.903918744 +0000 UTC m=+2688.620312435" watchObservedRunningTime="2026-07-14 18:17:11.908602427 +0000 UTC m=+2688.624996118"
Jul 14 18:17:13.087265 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:13.087225    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:17:14.895662 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:14.895565    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:17:21.160624 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:21.160583    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:21.161045 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:21.160638    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:21.177331 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:21.177298    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:21.952298 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:21.952268    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:17:24.422526 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:24.422498    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 18:17:24.428716 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:24.428696    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9hr5_e16a2359-cd0f-4b37-99d9-938bd35c13f7/ovn-acl-logging/0.log"
Jul 14 18:17:30.910450 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:30.910412    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:17:31.017961 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:31.017931    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:17:31.903325 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:31.903293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:17:37.904786 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:37.904741    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:17:38.928632 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:38.928594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:17:39.029129 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:39.029097    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:17:53.503869 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:17:53.503833    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:18:00.039733 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:00.039702    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"]
Jul 14 18:18:00.059320 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:00.059285    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:18:01.117732 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:01.117703    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"]
Jul 14 18:18:01.138163 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:01.138128    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:18:02.042828 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:02.042792    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7664-140676642438960"]
Jul 14 18:18:02.046514 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:02.046497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642438960"
Jul 14 18:18:02.049062 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:02.049044    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s5mzc\""
Jul 14 18:18:02.054152 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:02.054132    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140676642438960"]
Jul 14 18:18:02.126879 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:02.126830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn5gj\" (UniqueName: \"kubernetes.io/projected/91f45272-2456-45cb-bb65-a2952ecb1c64-kube-api-access-gn5gj\") pod \"test-curl-7664-140676642438960\" (UID: \"91f45272-2456-45cb-bb65-a2952ecb1c64\") " pod="opendatahub/test-curl-7664-140676642438960"
Jul 14 18:18:02.227745 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:02.227710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn5gj\" (UniqueName: \"kubernetes.io/projected/91f45272-2456-45cb-bb65-a2952ecb1c64-kube-api-access-gn5gj\") pod \"test-curl-7664-140676642438960\" (UID: \"91f45272-2456-45cb-bb65-a2952ecb1c64\") " pod="opendatahub/test-curl-7664-140676642438960"
Jul 14 18:18:02.237794 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:02.237760    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn5gj\" (UniqueName: \"kubernetes.io/projected/91f45272-2456-45cb-bb65-a2952ecb1c64-kube-api-access-gn5gj\") pod \"test-curl-7664-140676642438960\" (UID: \"91f45272-2456-45cb-bb65-a2952ecb1c64\") " pod="opendatahub/test-curl-7664-140676642438960"
Jul 14 18:18:02.355948 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:02.355877    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642438960"
Jul 14 18:18:02.498109 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:02.498085    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140676642438960"]
Jul 14 18:18:02.499472 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:18:02.499445    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91f45272_2456_45cb_bb65_a2952ecb1c64.slice/crio-e9801c5282ac15ca26c25037be2e3ee822b1099568323123a3b4e664ae76a3a7 WatchSource:0}: Error finding container e9801c5282ac15ca26c25037be2e3ee822b1099568323123a3b4e664ae76a3a7: Status 404 returned error can't find the container with id e9801c5282ac15ca26c25037be2e3ee822b1099568323123a3b4e664ae76a3a7
Jul 14 18:18:02.500939 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:02.500921    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 18:18:03.192944 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:03.192911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140676642438960" event={"ID":"91f45272-2456-45cb-bb65-a2952ecb1c64","Type":"ContainerStarted","Data":"e9801c5282ac15ca26c25037be2e3ee822b1099568323123a3b4e664ae76a3a7"}
Jul 14 18:18:04.200226 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:04.200173    2578 generic.go:356] "Generic (PLEG): container finished" podID="91f45272-2456-45cb-bb65-a2952ecb1c64" containerID="62b435fcccd1f15aea7b022933a4927f625652fe5259ac661525848f901fda7a" exitCode=0
Jul 14 18:18:04.200612 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:04.200238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140676642438960" event={"ID":"91f45272-2456-45cb-bb65-a2952ecb1c64","Type":"ContainerDied","Data":"62b435fcccd1f15aea7b022933a4927f625652fe5259ac661525848f901fda7a"}
Jul 14 18:18:05.347627 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.347604    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642438960"
Jul 14 18:18:05.371649 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.371616    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7664-140676642438960_91f45272-2456-45cb-bb65-a2952ecb1c64/test-curl-7664-140676642438960/0.log"
Jul 14 18:18:05.397918 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.397886    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7664-140676642438960"]
Jul 14 18:18:05.401876 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.401855    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7664-140676642438960"]
Jul 14 18:18:05.453477 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.453452    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn5gj\" (UniqueName: \"kubernetes.io/projected/91f45272-2456-45cb-bb65-a2952ecb1c64-kube-api-access-gn5gj\") pod \"91f45272-2456-45cb-bb65-a2952ecb1c64\" (UID: \"91f45272-2456-45cb-bb65-a2952ecb1c64\") "
Jul 14 18:18:05.455742 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.455719    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f45272-2456-45cb-bb65-a2952ecb1c64-kube-api-access-gn5gj" (OuterVolumeSpecName: "kube-api-access-gn5gj") pod "91f45272-2456-45cb-bb65-a2952ecb1c64" (UID: "91f45272-2456-45cb-bb65-a2952ecb1c64"). InnerVolumeSpecName "kube-api-access-gn5gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:18:05.523581 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.523548    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7664-140676642438832"]
Jul 14 18:18:05.523941 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.523929    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f45272-2456-45cb-bb65-a2952ecb1c64" containerName="test-curl-7664-140676642438960"
Jul 14 18:18:05.523984 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.523942    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f45272-2456-45cb-bb65-a2952ecb1c64" containerName="test-curl-7664-140676642438960"
Jul 14 18:18:05.524020 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.524013    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f45272-2456-45cb-bb65-a2952ecb1c64" containerName="test-curl-7664-140676642438960"
Jul 14 18:18:05.527274 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.527258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642438832"
Jul 14 18:18:05.536055 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.536030    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140676642438832"]
Jul 14 18:18:05.554575 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.554540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmf7z\" (UniqueName: \"kubernetes.io/projected/50a00979-fdce-48f3-94bf-4a8838ad61a4-kube-api-access-mmf7z\") pod \"test-curl-7664-140676642438832\" (UID: \"50a00979-fdce-48f3-94bf-4a8838ad61a4\") " pod="opendatahub/test-curl-7664-140676642438832"
Jul 14 18:18:05.554705 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.554652    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn5gj\" (UniqueName: \"kubernetes.io/projected/91f45272-2456-45cb-bb65-a2952ecb1c64-kube-api-access-gn5gj\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:05.655647 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.655613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmf7z\" (UniqueName: \"kubernetes.io/projected/50a00979-fdce-48f3-94bf-4a8838ad61a4-kube-api-access-mmf7z\") pod \"test-curl-7664-140676642438832\" (UID: \"50a00979-fdce-48f3-94bf-4a8838ad61a4\") " pod="opendatahub/test-curl-7664-140676642438832"
Jul 14 18:18:05.666623 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.666592    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmf7z\" (UniqueName: \"kubernetes.io/projected/50a00979-fdce-48f3-94bf-4a8838ad61a4-kube-api-access-mmf7z\") pod \"test-curl-7664-140676642438832\" (UID: \"50a00979-fdce-48f3-94bf-4a8838ad61a4\") " pod="opendatahub/test-curl-7664-140676642438832"
Jul 14 18:18:05.783303 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.783194    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91f45272-2456-45cb-bb65-a2952ecb1c64" path="/var/lib/kubelet/pods/91f45272-2456-45cb-bb65-a2952ecb1c64/volumes"
Jul 14 18:18:05.836770 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.836741    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642438832"
Jul 14 18:18:05.974580 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:05.974554    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140676642438832"]
Jul 14 18:18:05.976479 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:18:05.976456    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50a00979_fdce_48f3_94bf_4a8838ad61a4.slice/crio-b5b2f9577a244c51f1834ac254d48ef4eadef48399a64400da5f5b34ad3a252f WatchSource:0}: Error finding container b5b2f9577a244c51f1834ac254d48ef4eadef48399a64400da5f5b34ad3a252f: Status 404 returned error can't find the container with id b5b2f9577a244c51f1834ac254d48ef4eadef48399a64400da5f5b34ad3a252f
Jul 14 18:18:06.212948 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:06.212917    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642438960"
Jul 14 18:18:06.213139 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:06.213016    2578 scope.go:117] "RemoveContainer" containerID="62b435fcccd1f15aea7b022933a4927f625652fe5259ac661525848f901fda7a"
Jul 14 18:18:06.214418 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:06.214277    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140676642438832" event={"ID":"50a00979-fdce-48f3-94bf-4a8838ad61a4","Type":"ContainerStarted","Data":"b5b2f9577a244c51f1834ac254d48ef4eadef48399a64400da5f5b34ad3a252f"}
Jul 14 18:18:07.096280 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:07.096243    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:18:07.222477 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:07.222444    2578 generic.go:356] "Generic (PLEG): container finished" podID="50a00979-fdce-48f3-94bf-4a8838ad61a4" containerID="3520ee70cf457cfb5ba2fe5f61503760d92a91392f68c0e4e52aa5ec35a1b35a" exitCode=0
Jul 14 18:18:07.222651 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:07.222532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140676642438832" event={"ID":"50a00979-fdce-48f3-94bf-4a8838ad61a4","Type":"ContainerDied","Data":"3520ee70cf457cfb5ba2fe5f61503760d92a91392f68c0e4e52aa5ec35a1b35a"}
Jul 14 18:18:08.365396 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.365372    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642438832"
Jul 14 18:18:08.382921 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.382784    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmf7z\" (UniqueName: \"kubernetes.io/projected/50a00979-fdce-48f3-94bf-4a8838ad61a4-kube-api-access-mmf7z\") pod \"50a00979-fdce-48f3-94bf-4a8838ad61a4\" (UID: \"50a00979-fdce-48f3-94bf-4a8838ad61a4\") "
Jul 14 18:18:08.385420 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.385393    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a00979-fdce-48f3-94bf-4a8838ad61a4-kube-api-access-mmf7z" (OuterVolumeSpecName: "kube-api-access-mmf7z") pod "50a00979-fdce-48f3-94bf-4a8838ad61a4" (UID: "50a00979-fdce-48f3-94bf-4a8838ad61a4"). InnerVolumeSpecName "kube-api-access-mmf7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:18:08.389280 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.389240    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7664-140676642438832_50a00979-fdce-48f3-94bf-4a8838ad61a4/test-curl-7664-140676642438832/0.log"
Jul 14 18:18:08.418517 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.418493    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7664-140676642438832"]
Jul 14 18:18:08.423000 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.422979    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7664-140676642438832"]
Jul 14 18:18:08.484173 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.484097    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmf7z\" (UniqueName: \"kubernetes.io/projected/50a00979-fdce-48f3-94bf-4a8838ad61a4-kube-api-access-mmf7z\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:08.575576 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.575545    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7664-140676642469616"]
Jul 14 18:18:08.575952 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.575940    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a00979-fdce-48f3-94bf-4a8838ad61a4" containerName="test-curl-7664-140676642438832"
Jul 14 18:18:08.576005 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.575954    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a00979-fdce-48f3-94bf-4a8838ad61a4" containerName="test-curl-7664-140676642438832"
Jul 14 18:18:08.576038 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.576023    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="50a00979-fdce-48f3-94bf-4a8838ad61a4" containerName="test-curl-7664-140676642438832"
Jul 14 18:18:08.579034 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.579017    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:08.585928 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.585902    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140676642469616"]
Jul 14 18:18:08.686070 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.686035    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbrk5\" (UniqueName: \"kubernetes.io/projected/d0a46169-6ab8-431a-8710-7a5dd50f1641-kube-api-access-zbrk5\") pod \"test-curl-7664-140676642469616\" (UID: \"d0a46169-6ab8-431a-8710-7a5dd50f1641\") " pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:08.786753 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.786667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbrk5\" (UniqueName: \"kubernetes.io/projected/d0a46169-6ab8-431a-8710-7a5dd50f1641-kube-api-access-zbrk5\") pod \"test-curl-7664-140676642469616\" (UID: \"d0a46169-6ab8-431a-8710-7a5dd50f1641\") " pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:08.796797 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.796767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbrk5\" (UniqueName: \"kubernetes.io/projected/d0a46169-6ab8-431a-8710-7a5dd50f1641-kube-api-access-zbrk5\") pod \"test-curl-7664-140676642469616\" (UID: \"d0a46169-6ab8-431a-8710-7a5dd50f1641\") " pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:08.888411 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:08.888383    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:09.026821 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:09.026795    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140676642469616"]
Jul 14 18:18:09.028729 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:18:09.028699    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0a46169_6ab8_431a_8710_7a5dd50f1641.slice/crio-4f8fb16de3a3aada5bcddf4035367b631d000239fc032e20918d22bb6d52f0e9 WatchSource:0}: Error finding container 4f8fb16de3a3aada5bcddf4035367b631d000239fc032e20918d22bb6d52f0e9: Status 404 returned error can't find the container with id 4f8fb16de3a3aada5bcddf4035367b631d000239fc032e20918d22bb6d52f0e9
Jul 14 18:18:09.235939 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:09.235908    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642438832"
Jul 14 18:18:09.235939 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:09.235937    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5b2f9577a244c51f1834ac254d48ef4eadef48399a64400da5f5b34ad3a252f"
Jul 14 18:18:09.237128 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:09.237096    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140676642469616" event={"ID":"d0a46169-6ab8-431a-8710-7a5dd50f1641","Type":"ContainerStarted","Data":"4f8fb16de3a3aada5bcddf4035367b631d000239fc032e20918d22bb6d52f0e9"}
Jul 14 18:18:09.246539 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:18:09.246512    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7664-140676642438832\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="50a00979-fdce-48f3-94bf-4a8838ad61a4" pod="opendatahub/test-curl-7664-140676642438832"
Jul 14 18:18:09.783452 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:09.783423    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50a00979-fdce-48f3-94bf-4a8838ad61a4" path="/var/lib/kubelet/pods/50a00979-fdce-48f3-94bf-4a8838ad61a4/volumes"
Jul 14 18:18:10.244480 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:10.244440    2578 generic.go:356] "Generic (PLEG): container finished" podID="d0a46169-6ab8-431a-8710-7a5dd50f1641" containerID="d80c9f6ccd72865128308d87d141f484c8ee205a305bef4d3b20a32cb75d7e9a" exitCode=0
Jul 14 18:18:10.244660 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:10.244530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140676642469616" event={"ID":"d0a46169-6ab8-431a-8710-7a5dd50f1641","Type":"ContainerDied","Data":"d80c9f6ccd72865128308d87d141f484c8ee205a305bef4d3b20a32cb75d7e9a"}
Jul 14 18:18:11.394793 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.394766    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:11.421025 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.420996    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7664-140676642469616_d0a46169-6ab8-431a-8710-7a5dd50f1641/test-curl-7664-140676642469616/0.log"
Jul 14 18:18:11.458560 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.458532    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7664-140676642469616"]
Jul 14 18:18:11.464305 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.464279    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7664-140676642469616"]
Jul 14 18:18:11.513094 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.513021    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbrk5\" (UniqueName: \"kubernetes.io/projected/d0a46169-6ab8-431a-8710-7a5dd50f1641-kube-api-access-zbrk5\") pod \"d0a46169-6ab8-431a-8710-7a5dd50f1641\" (UID: \"d0a46169-6ab8-431a-8710-7a5dd50f1641\") "
Jul 14 18:18:11.515345 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.515311    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a46169-6ab8-431a-8710-7a5dd50f1641-kube-api-access-zbrk5" (OuterVolumeSpecName: "kube-api-access-zbrk5") pod "d0a46169-6ab8-431a-8710-7a5dd50f1641" (UID: "d0a46169-6ab8-431a-8710-7a5dd50f1641"). InnerVolumeSpecName "kube-api-access-zbrk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:18:11.584049 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.584016    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7664-140676642469616"]
Jul 14 18:18:11.584457 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.584444    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a46169-6ab8-431a-8710-7a5dd50f1641" containerName="test-curl-7664-140676642469616"
Jul 14 18:18:11.584507 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.584459    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a46169-6ab8-431a-8710-7a5dd50f1641" containerName="test-curl-7664-140676642469616"
Jul 14 18:18:11.584544 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.584539    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a46169-6ab8-431a-8710-7a5dd50f1641" containerName="test-curl-7664-140676642469616"
Jul 14 18:18:11.587547 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.587531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:11.596952 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.596926    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140676642469616"]
Jul 14 18:18:11.613968 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.613940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42bfq\" (UniqueName: \"kubernetes.io/projected/bbfc674d-4044-4b36-bd0e-e0f0b6c54da8-kube-api-access-42bfq\") pod \"test-curl-7664-140676642469616\" (UID: \"bbfc674d-4044-4b36-bd0e-e0f0b6c54da8\") " pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:11.614107 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.614015    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbrk5\" (UniqueName: \"kubernetes.io/projected/d0a46169-6ab8-431a-8710-7a5dd50f1641-kube-api-access-zbrk5\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:11.714676 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.714638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42bfq\" (UniqueName: \"kubernetes.io/projected/bbfc674d-4044-4b36-bd0e-e0f0b6c54da8-kube-api-access-42bfq\") pod \"test-curl-7664-140676642469616\" (UID: \"bbfc674d-4044-4b36-bd0e-e0f0b6c54da8\") " pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:11.725315 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.725283    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42bfq\" (UniqueName: \"kubernetes.io/projected/bbfc674d-4044-4b36-bd0e-e0f0b6c54da8-kube-api-access-42bfq\") pod \"test-curl-7664-140676642469616\" (UID: \"bbfc674d-4044-4b36-bd0e-e0f0b6c54da8\") " pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:11.784422 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.784351    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0a46169-6ab8-431a-8710-7a5dd50f1641" path="/var/lib/kubelet/pods/d0a46169-6ab8-431a-8710-7a5dd50f1641/volumes"
Jul 14 18:18:11.897087 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:11.897054    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:12.042529 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:12.042500    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140676642469616"]
Jul 14 18:18:12.043837 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:18:12.043804    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbfc674d_4044_4b36_bd0e_e0f0b6c54da8.slice/crio-255daeb1e2814022950493b9f1cc2b2c415b1c16e12c455b518caa2aea73b63a WatchSource:0}: Error finding container 255daeb1e2814022950493b9f1cc2b2c415b1c16e12c455b518caa2aea73b63a: Status 404 returned error can't find the container with id 255daeb1e2814022950493b9f1cc2b2c415b1c16e12c455b518caa2aea73b63a
Jul 14 18:18:12.257880 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:12.257854    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:12.258070 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:12.257911    2578 scope.go:117] "RemoveContainer" containerID="d80c9f6ccd72865128308d87d141f484c8ee205a305bef4d3b20a32cb75d7e9a"
Jul 14 18:18:12.259389 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:12.259083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140676642469616" event={"ID":"bbfc674d-4044-4b36-bd0e-e0f0b6c54da8","Type":"ContainerStarted","Data":"255daeb1e2814022950493b9f1cc2b2c415b1c16e12c455b518caa2aea73b63a"}
Jul 14 18:18:13.268072 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:13.268037    2578 generic.go:356] "Generic (PLEG): container finished" podID="bbfc674d-4044-4b36-bd0e-e0f0b6c54da8" containerID="dcdd54c52011512ff8ca5a4b714b6144d1c58f51faa0c6c9116e3bbb21a9dc89" exitCode=0
Jul 14 18:18:13.268501 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:13.268127    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140676642469616" event={"ID":"bbfc674d-4044-4b36-bd0e-e0f0b6c54da8","Type":"ContainerDied","Data":"dcdd54c52011512ff8ca5a4b714b6144d1c58f51faa0c6c9116e3bbb21a9dc89"}
Jul 14 18:18:14.410140 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.410115    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:14.454683 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.454655    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7664-140676642469616_bbfc674d-4044-4b36-bd0e-e0f0b6c54da8/test-curl-7664-140676642469616/0.log"
Jul 14 18:18:14.500754 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.500727    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7664-140676642469616"]
Jul 14 18:18:14.511089 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.511063    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7664-140676642469616"]
Jul 14 18:18:14.539844 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.539749    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42bfq\" (UniqueName: \"kubernetes.io/projected/bbfc674d-4044-4b36-bd0e-e0f0b6c54da8-kube-api-access-42bfq\") pod \"bbfc674d-4044-4b36-bd0e-e0f0b6c54da8\" (UID: \"bbfc674d-4044-4b36-bd0e-e0f0b6c54da8\") "
Jul 14 18:18:14.541928 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.541895    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfc674d-4044-4b36-bd0e-e0f0b6c54da8-kube-api-access-42bfq" (OuterVolumeSpecName: "kube-api-access-42bfq") pod "bbfc674d-4044-4b36-bd0e-e0f0b6c54da8" (UID: "bbfc674d-4044-4b36-bd0e-e0f0b6c54da8"). InnerVolumeSpecName "kube-api-access-42bfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:18:14.639300 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.639269    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7664-140676642470624"]
Jul 14 18:18:14.639667 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.639654    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbfc674d-4044-4b36-bd0e-e0f0b6c54da8" containerName="test-curl-7664-140676642469616"
Jul 14 18:18:14.639714 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.639668    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbfc674d-4044-4b36-bd0e-e0f0b6c54da8" containerName="test-curl-7664-140676642469616"
Jul 14 18:18:14.639763 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.639750    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbfc674d-4044-4b36-bd0e-e0f0b6c54da8" containerName="test-curl-7664-140676642469616"
Jul 14 18:18:14.640790 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.640766    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42bfq\" (UniqueName: \"kubernetes.io/projected/bbfc674d-4044-4b36-bd0e-e0f0b6c54da8-kube-api-access-42bfq\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:14.642922 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.642906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642470624"
Jul 14 18:18:14.656887 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.656862    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140676642470624"]
Jul 14 18:18:14.741262 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.741191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzflb\" (UniqueName: \"kubernetes.io/projected/44681355-ae6a-48c5-8342-b181513b4ad6-kube-api-access-rzflb\") pod \"test-curl-7664-140676642470624\" (UID: \"44681355-ae6a-48c5-8342-b181513b4ad6\") " pod="opendatahub/test-curl-7664-140676642470624"
Jul 14 18:18:14.842179 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.842079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzflb\" (UniqueName: \"kubernetes.io/projected/44681355-ae6a-48c5-8342-b181513b4ad6-kube-api-access-rzflb\") pod \"test-curl-7664-140676642470624\" (UID: \"44681355-ae6a-48c5-8342-b181513b4ad6\") " pod="opendatahub/test-curl-7664-140676642470624"
Jul 14 18:18:14.853231 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.853173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzflb\" (UniqueName: \"kubernetes.io/projected/44681355-ae6a-48c5-8342-b181513b4ad6-kube-api-access-rzflb\") pod \"test-curl-7664-140676642470624\" (UID: \"44681355-ae6a-48c5-8342-b181513b4ad6\") " pod="opendatahub/test-curl-7664-140676642470624"
Jul 14 18:18:14.952015 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:14.951985    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642470624"
Jul 14 18:18:15.064045 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.063998    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg" podUID="d3d0081d-6030-4bef-908a-269f6f90cd95" containerName="main" containerID="cri-o://d77bb1b693519fad62e1c079bc0c78560b93110dbc3f389cdb4d2bde945a65ca" gracePeriod=45
Jul 14 18:18:15.100751 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.100724    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140676642470624"]
Jul 14 18:18:15.125149 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:18:15.125101    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44681355_ae6a_48c5_8342_b181513b4ad6.slice/crio-7b1986c3c9754aa00d7d88d1b5b3d2042e2e9c1b75d55c2e819c79752b58f2de WatchSource:0}: Error finding container 7b1986c3c9754aa00d7d88d1b5b3d2042e2e9c1b75d55c2e819c79752b58f2de: Status 404 returned error can't find the container with id 7b1986c3c9754aa00d7d88d1b5b3d2042e2e9c1b75d55c2e819c79752b58f2de
Jul 14 18:18:15.282614 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.282578    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140676642470624" event={"ID":"44681355-ae6a-48c5-8342-b181513b4ad6","Type":"ContainerStarted","Data":"7b1986c3c9754aa00d7d88d1b5b3d2042e2e9c1b75d55c2e819c79752b58f2de"}
Jul 14 18:18:15.284395 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.284189    2578 generic.go:356] "Generic (PLEG): container finished" podID="d3d0081d-6030-4bef-908a-269f6f90cd95" containerID="d77bb1b693519fad62e1c079bc0c78560b93110dbc3f389cdb4d2bde945a65ca" exitCode=0
Jul 14 18:18:15.284557 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.284248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg" event={"ID":"d3d0081d-6030-4bef-908a-269f6f90cd95","Type":"ContainerDied","Data":"d77bb1b693519fad62e1c079bc0c78560b93110dbc3f389cdb4d2bde945a65ca"}
Jul 14 18:18:15.285853 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.285833    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:15.285853 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.285849    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="255daeb1e2814022950493b9f1cc2b2c415b1c16e12c455b518caa2aea73b63a"
Jul 14 18:18:15.296160 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:18:15.296133    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7664-140676642469616\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="bbfc674d-4044-4b36-bd0e-e0f0b6c54da8" pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:15.306667 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.306643    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:18:15.313215 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:18:15.313176    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7664-140676642469616\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="bbfc674d-4044-4b36-bd0e-e0f0b6c54da8" pod="opendatahub/test-curl-7664-140676642469616"
Jul 14 18:18:15.346529 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.346494    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-dshm\") pod \"d3d0081d-6030-4bef-908a-269f6f90cd95\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") "
Jul 14 18:18:15.346529 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.346533    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-model-cache\") pod \"d3d0081d-6030-4bef-908a-269f6f90cd95\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") "
Jul 14 18:18:15.346729 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.346568    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fgzk\" (UniqueName: \"kubernetes.io/projected/d3d0081d-6030-4bef-908a-269f6f90cd95-kube-api-access-9fgzk\") pod \"d3d0081d-6030-4bef-908a-269f6f90cd95\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") "
Jul 14 18:18:15.346729 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.346632    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d0081d-6030-4bef-908a-269f6f90cd95-tls-certs\") pod \"d3d0081d-6030-4bef-908a-269f6f90cd95\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") "
Jul 14 18:18:15.346729 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.346655    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-tmp-dir\") pod \"d3d0081d-6030-4bef-908a-269f6f90cd95\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") "
Jul 14 18:18:15.346914 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.346754    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-home\") pod \"d3d0081d-6030-4bef-908a-269f6f90cd95\" (UID: \"d3d0081d-6030-4bef-908a-269f6f90cd95\") "
Jul 14 18:18:15.346914 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.346824    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3d0081d-6030-4bef-908a-269f6f90cd95" (UID: "d3d0081d-6030-4bef-908a-269f6f90cd95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:18:15.347037 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.347007    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3d0081d-6030-4bef-908a-269f6f90cd95" (UID: "d3d0081d-6030-4bef-908a-269f6f90cd95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:18:15.347109 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.347046    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-home" (OuterVolumeSpecName: "home") pod "d3d0081d-6030-4bef-908a-269f6f90cd95" (UID: "d3d0081d-6030-4bef-908a-269f6f90cd95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:18:15.347156 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.347130    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:15.347156 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.347148    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:15.347243 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.347159    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:15.348867 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.348847    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-dshm" (OuterVolumeSpecName: "dshm") pod "d3d0081d-6030-4bef-908a-269f6f90cd95" (UID: "d3d0081d-6030-4bef-908a-269f6f90cd95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:18:15.348956 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.348883    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3d0081d-6030-4bef-908a-269f6f90cd95-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3d0081d-6030-4bef-908a-269f6f90cd95" (UID: "d3d0081d-6030-4bef-908a-269f6f90cd95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:18:15.348956 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.348913    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d0081d-6030-4bef-908a-269f6f90cd95-kube-api-access-9fgzk" (OuterVolumeSpecName: "kube-api-access-9fgzk") pod "d3d0081d-6030-4bef-908a-269f6f90cd95" (UID: "d3d0081d-6030-4bef-908a-269f6f90cd95"). InnerVolumeSpecName "kube-api-access-9fgzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:18:15.448563 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.448526    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d0081d-6030-4bef-908a-269f6f90cd95-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:15.448563 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.448557    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d0081d-6030-4bef-908a-269f6f90cd95-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:15.448563 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.448568    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fgzk\" (UniqueName: \"kubernetes.io/projected/d3d0081d-6030-4bef-908a-269f6f90cd95-kube-api-access-9fgzk\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:15.783845 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:15.783811    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbfc674d-4044-4b36-bd0e-e0f0b6c54da8" path="/var/lib/kubelet/pods/bbfc674d-4044-4b36-bd0e-e0f0b6c54da8/volumes"
Jul 14 18:18:16.138711 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.138624    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr" podUID="ddacd2e0-5ad5-46b1-9825-9f9df528d1cb" containerName="main" containerID="cri-o://4d831200227b4fae9d3f2754832955cd6b82fefbc55b393de3e602664e2117a4" gracePeriod=45
Jul 14 18:18:16.295003 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.294977    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"
Jul 14 18:18:16.295171 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.295014    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg" event={"ID":"d3d0081d-6030-4bef-908a-269f6f90cd95","Type":"ContainerDied","Data":"09b0aa3805c6bf620dda62620a4d372427fbf31b2889235cb4399964d50f176d"}
Jul 14 18:18:16.295171 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.295072    2578 scope.go:117] "RemoveContainer" containerID="d77bb1b693519fad62e1c079bc0c78560b93110dbc3f389cdb4d2bde945a65ca"
Jul 14 18:18:16.297094 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.297065    2578 generic.go:356] "Generic (PLEG): container finished" podID="ddacd2e0-5ad5-46b1-9825-9f9df528d1cb" containerID="4d831200227b4fae9d3f2754832955cd6b82fefbc55b393de3e602664e2117a4" exitCode=0
Jul 14 18:18:16.297279 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.297228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr" event={"ID":"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb","Type":"ContainerDied","Data":"4d831200227b4fae9d3f2754832955cd6b82fefbc55b393de3e602664e2117a4"}
Jul 14 18:18:16.299354 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.299325    2578 generic.go:356] "Generic (PLEG): container finished" podID="44681355-ae6a-48c5-8342-b181513b4ad6" containerID="767f74928efd41631b6897324ab6911f5aac1d2b59b557342b077ddee6dc1bf6" exitCode=0
Jul 14 18:18:16.299445 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.299365    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140676642470624" event={"ID":"44681355-ae6a-48c5-8342-b181513b4ad6","Type":"ContainerDied","Data":"767f74928efd41631b6897324ab6911f5aac1d2b59b557342b077ddee6dc1bf6"}
Jul 14 18:18:16.344582 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.344545    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"]
Jul 14 18:18:16.349172 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.349138    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-aa324e/rate-test-model-aa324e-kserve-788f6d97dc-m55vg"]
Jul 14 18:18:16.414156 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.414127    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:18:16.457492 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.457465    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr5tn\" (UniqueName: \"kubernetes.io/projected/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-kube-api-access-dr5tn\") pod \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") "
Jul 14 18:18:16.457492 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.457497    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-dshm\") pod \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") "
Jul 14 18:18:16.458017 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.457525    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-home\") pod \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") "
Jul 14 18:18:16.458017 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.457548    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-tls-certs\") pod \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") "
Jul 14 18:18:16.458017 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.457580    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-model-cache\") pod \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") "
Jul 14 18:18:16.458017 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.457620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-tmp-dir\") pod \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\" (UID: \"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb\") "
Jul 14 18:18:16.458017 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.457793    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-home" (OuterVolumeSpecName: "home") pod "ddacd2e0-5ad5-46b1-9825-9f9df528d1cb" (UID: "ddacd2e0-5ad5-46b1-9825-9f9df528d1cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:18:16.458017 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.457840    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "ddacd2e0-5ad5-46b1-9825-9f9df528d1cb" (UID: "ddacd2e0-5ad5-46b1-9825-9f9df528d1cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:18:16.458017 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.457994    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ddacd2e0-5ad5-46b1-9825-9f9df528d1cb" (UID: "ddacd2e0-5ad5-46b1-9825-9f9df528d1cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:18:16.458017 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.458001    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:16.458017 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.458021    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:16.459881 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.459847    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-dshm" (OuterVolumeSpecName: "dshm") pod "ddacd2e0-5ad5-46b1-9825-9f9df528d1cb" (UID: "ddacd2e0-5ad5-46b1-9825-9f9df528d1cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:18:16.459881 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.459867    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ddacd2e0-5ad5-46b1-9825-9f9df528d1cb" (UID: "ddacd2e0-5ad5-46b1-9825-9f9df528d1cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:18:16.460020 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.459948    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-kube-api-access-dr5tn" (OuterVolumeSpecName: "kube-api-access-dr5tn") pod "ddacd2e0-5ad5-46b1-9825-9f9df528d1cb" (UID: "ddacd2e0-5ad5-46b1-9825-9f9df528d1cb"). InnerVolumeSpecName "kube-api-access-dr5tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:18:16.559343 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.559302    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr5tn\" (UniqueName: \"kubernetes.io/projected/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-kube-api-access-dr5tn\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:16.559343 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.559332    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:16.559343 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.559342    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:16.559343 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:16.559351    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:17.307760 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.307727    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"
Jul 14 18:18:17.307760 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.307746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr" event={"ID":"ddacd2e0-5ad5-46b1-9825-9f9df528d1cb","Type":"ContainerDied","Data":"74658e091e0467124f2f1e31058812cfe7172b685f5093e0cc226746be06824c"}
Jul 14 18:18:17.308029 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.307807    2578 scope.go:117] "RemoveContainer" containerID="4d831200227b4fae9d3f2754832955cd6b82fefbc55b393de3e602664e2117a4"
Jul 14 18:18:17.339068 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.339040    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"]
Jul 14 18:18:17.341970 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.341939    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-877bd7/rate-test-model-877bd7-kserve-85b8bcb99f-fxwbr"]
Jul 14 18:18:17.449193 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.449172    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642470624"
Jul 14 18:18:17.471416 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.471393    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7664-140676642470624_44681355-ae6a-48c5-8342-b181513b4ad6/test-curl-7664-140676642470624/0.log"
Jul 14 18:18:17.496520 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.496491    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7664-140676642470624"]
Jul 14 18:18:17.501114 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.501090    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7664-140676642470624"]
Jul 14 18:18:17.568617 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.568534    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzflb\" (UniqueName: \"kubernetes.io/projected/44681355-ae6a-48c5-8342-b181513b4ad6-kube-api-access-rzflb\") pod \"44681355-ae6a-48c5-8342-b181513b4ad6\" (UID: \"44681355-ae6a-48c5-8342-b181513b4ad6\") "
Jul 14 18:18:17.570800 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.570779    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44681355-ae6a-48c5-8342-b181513b4ad6-kube-api-access-rzflb" (OuterVolumeSpecName: "kube-api-access-rzflb") pod "44681355-ae6a-48c5-8342-b181513b4ad6" (UID: "44681355-ae6a-48c5-8342-b181513b4ad6"). InnerVolumeSpecName "kube-api-access-rzflb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:18:17.627190 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.627154    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7664-140676642470624"]
Jul 14 18:18:17.627657 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.627642    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddacd2e0-5ad5-46b1-9825-9f9df528d1cb" containerName="main"
Jul 14 18:18:17.627702 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.627660    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddacd2e0-5ad5-46b1-9825-9f9df528d1cb" containerName="main"
Jul 14 18:18:17.627702 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.627681    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44681355-ae6a-48c5-8342-b181513b4ad6" containerName="test-curl-7664-140676642470624"
Jul 14 18:18:17.627702 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.627689    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="44681355-ae6a-48c5-8342-b181513b4ad6" containerName="test-curl-7664-140676642470624"
Jul 14 18:18:17.627702 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.627697    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3d0081d-6030-4bef-908a-269f6f90cd95" containerName="main"
Jul 14 18:18:17.627702 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.627702    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3d0081d-6030-4bef-908a-269f6f90cd95" containerName="main"
Jul 14 18:18:17.627860 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.627774    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3d0081d-6030-4bef-908a-269f6f90cd95" containerName="main"
Jul 14 18:18:17.627860 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.627782    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddacd2e0-5ad5-46b1-9825-9f9df528d1cb" containerName="main"
Jul 14 18:18:17.627860 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.627791    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="44681355-ae6a-48c5-8342-b181513b4ad6" containerName="test-curl-7664-140676642470624"
Jul 14 18:18:17.630960 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.630938    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642470624"
Jul 14 18:18:17.636861 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.636835    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140676642470624"]
Jul 14 18:18:17.670143 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.670109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm8qg\" (UniqueName: \"kubernetes.io/projected/ddec891a-8858-48a0-befb-238c513e00c5-kube-api-access-qm8qg\") pod \"test-curl-7664-140676642470624\" (UID: \"ddec891a-8858-48a0-befb-238c513e00c5\") " pod="opendatahub/test-curl-7664-140676642470624"
Jul 14 18:18:17.670336 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.670166    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzflb\" (UniqueName: \"kubernetes.io/projected/44681355-ae6a-48c5-8342-b181513b4ad6-kube-api-access-rzflb\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:17.771543 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.771496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm8qg\" (UniqueName: \"kubernetes.io/projected/ddec891a-8858-48a0-befb-238c513e00c5-kube-api-access-qm8qg\") pod \"test-curl-7664-140676642470624\" (UID: \"ddec891a-8858-48a0-befb-238c513e00c5\") " pod="opendatahub/test-curl-7664-140676642470624"
Jul 14 18:18:17.782507 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.782478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm8qg\" (UniqueName: \"kubernetes.io/projected/ddec891a-8858-48a0-befb-238c513e00c5-kube-api-access-qm8qg\") pod \"test-curl-7664-140676642470624\" (UID: \"ddec891a-8858-48a0-befb-238c513e00c5\") " pod="opendatahub/test-curl-7664-140676642470624"
Jul 14 18:18:17.783413 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.783391    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44681355-ae6a-48c5-8342-b181513b4ad6" path="/var/lib/kubelet/pods/44681355-ae6a-48c5-8342-b181513b4ad6/volumes"
Jul 14 18:18:17.783774 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.783762    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3d0081d-6030-4bef-908a-269f6f90cd95" path="/var/lib/kubelet/pods/d3d0081d-6030-4bef-908a-269f6f90cd95/volumes"
Jul 14 18:18:17.784099 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.784087    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddacd2e0-5ad5-46b1-9825-9f9df528d1cb" path="/var/lib/kubelet/pods/ddacd2e0-5ad5-46b1-9825-9f9df528d1cb/volumes"
Jul 14 18:18:17.941000 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:17.940961    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642470624"
Jul 14 18:18:18.086022 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:18.085995    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140676642470624"]
Jul 14 18:18:18.086801 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:18:18.086774    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddec891a_8858_48a0_befb_238c513e00c5.slice/crio-91748224dd6bb17c0878b136b62dd1b237e91b5517bec47ea69bd679e050e957 WatchSource:0}: Error finding container 91748224dd6bb17c0878b136b62dd1b237e91b5517bec47ea69bd679e050e957: Status 404 returned error can't find the container with id 91748224dd6bb17c0878b136b62dd1b237e91b5517bec47ea69bd679e050e957
Jul 14 18:18:18.315805 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:18.315728    2578 scope.go:117] "RemoveContainer" containerID="767f74928efd41631b6897324ab6911f5aac1d2b59b557342b077ddee6dc1bf6"
Jul 14 18:18:18.315805 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:18.315729    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642470624"
Jul 14 18:18:18.317059 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:18.317026    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140676642470624" event={"ID":"ddec891a-8858-48a0-befb-238c513e00c5","Type":"ContainerStarted","Data":"91748224dd6bb17c0878b136b62dd1b237e91b5517bec47ea69bd679e050e957"}
Jul 14 18:18:19.328111 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:19.328079    2578 generic.go:356] "Generic (PLEG): container finished" podID="ddec891a-8858-48a0-befb-238c513e00c5" containerID="93a57fbc2e6bfaf60a1a2fb6a483f445a242483c4a6d2ceeb71e55028d5efdc0" exitCode=0
Jul 14 18:18:19.328497 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:19.328228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140676642470624" event={"ID":"ddec891a-8858-48a0-befb-238c513e00c5","Type":"ContainerDied","Data":"93a57fbc2e6bfaf60a1a2fb6a483f445a242483c4a6d2ceeb71e55028d5efdc0"}
Jul 14 18:18:20.470388 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.470367    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642470624"
Jul 14 18:18:20.494167 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.494146    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7664-140676642470624_ddec891a-8858-48a0-befb-238c513e00c5/test-curl-7664-140676642470624/0.log"
Jul 14 18:18:20.519464 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.519436    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7664-140676642470624"]
Jul 14 18:18:20.523286 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.523261    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7664-140676642470624"]
Jul 14 18:18:20.595843 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.595765    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm8qg\" (UniqueName: \"kubernetes.io/projected/ddec891a-8858-48a0-befb-238c513e00c5-kube-api-access-qm8qg\") pod \"ddec891a-8858-48a0-befb-238c513e00c5\" (UID: \"ddec891a-8858-48a0-befb-238c513e00c5\") "
Jul 14 18:18:20.597919 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.597894    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddec891a-8858-48a0-befb-238c513e00c5-kube-api-access-qm8qg" (OuterVolumeSpecName: "kube-api-access-qm8qg") pod "ddec891a-8858-48a0-befb-238c513e00c5" (UID: "ddec891a-8858-48a0-befb-238c513e00c5"). InnerVolumeSpecName "kube-api-access-qm8qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:18:20.696762 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.696730    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm8qg\" (UniqueName: \"kubernetes.io/projected/ddec891a-8858-48a0-befb-238c513e00c5-kube-api-access-qm8qg\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:20.714453 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.714423    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc"]
Jul 14 18:18:20.714845 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.714832    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddec891a-8858-48a0-befb-238c513e00c5" containerName="test-curl-7664-140676642470624"
Jul 14 18:18:20.714910 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.714848    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddec891a-8858-48a0-befb-238c513e00c5" containerName="test-curl-7664-140676642470624"
Jul 14 18:18:20.714945 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.714927    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddec891a-8858-48a0-befb-238c513e00c5" containerName="test-curl-7664-140676642470624"
Jul 14 18:18:20.718023 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.718007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc"
Jul 14 18:18:20.720601 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.720587    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pd7w9\""
Jul 14 18:18:20.730691 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.729107    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc"]
Jul 14 18:18:20.797403 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.797366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/738b42b2-5340-4570-b696-d5de9540c6ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc\" (UID: \"738b42b2-5340-4570-b696-d5de9540c6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc"
Jul 14 18:18:20.897892 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.897794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/738b42b2-5340-4570-b696-d5de9540c6ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc\" (UID: \"738b42b2-5340-4570-b696-d5de9540c6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc"
Jul 14 18:18:20.898493 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:20.898467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/738b42b2-5340-4570-b696-d5de9540c6ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc\" (UID: \"738b42b2-5340-4570-b696-d5de9540c6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc"
Jul 14 18:18:21.035942 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:21.035904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc"
Jul 14 18:18:21.173510 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:21.173484    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc"]
Jul 14 18:18:21.174651 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:18:21.174624    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738b42b2_5340_4570_b696_d5de9540c6ea.slice/crio-f30dce653f6ec73f19df90d5b306c8707f9b2bed7368d7d8025e19dfe52d27bf WatchSource:0}: Error finding container f30dce653f6ec73f19df90d5b306c8707f9b2bed7368d7d8025e19dfe52d27bf: Status 404 returned error can't find the container with id f30dce653f6ec73f19df90d5b306c8707f9b2bed7368d7d8025e19dfe52d27bf
Jul 14 18:18:21.341006 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:21.340976    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:18:21.343004 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:21.342978    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91748224dd6bb17c0878b136b62dd1b237e91b5517bec47ea69bd679e050e957"
Jul 14 18:18:21.343004 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:21.342991    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140676642470624"
Jul 14 18:18:21.344242 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:21.344200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc" event={"ID":"738b42b2-5340-4570-b696-d5de9540c6ea","Type":"ContainerStarted","Data":"f30dce653f6ec73f19df90d5b306c8707f9b2bed7368d7d8025e19dfe52d27bf"}
Jul 14 18:18:21.353034 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:18:21.353006    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7664-140676642470624\" is forbidden: User \"system:node:ip-10-0-135-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-231.ec2.internal' and this object" podUID="ddec891a-8858-48a0-befb-238c513e00c5" pod="opendatahub/test-curl-7664-140676642470624"
Jul 14 18:18:21.448810 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:21.448728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:18:21.785099 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:21.785027    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddec891a-8858-48a0-befb-238c513e00c5" path="/var/lib/kubelet/pods/ddec891a-8858-48a0-befb-238c513e00c5/volumes"
Jul 14 18:18:22.354037 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:22.353994    2578 generic.go:356] "Generic (PLEG): container finished" podID="738b42b2-5340-4570-b696-d5de9540c6ea" containerID="6b74f47d4306c8f5b1cca9515d33b12f9cca715572a135ddb08ffde66a43c1a5" exitCode=0
Jul 14 18:18:22.354238 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:22.354073    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc" event={"ID":"738b42b2-5340-4570-b696-d5de9540c6ea","Type":"ContainerDied","Data":"6b74f47d4306c8f5b1cca9515d33b12f9cca715572a135ddb08ffde66a43c1a5"}
Jul 14 18:18:23.500050 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:23.500029    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc"
Jul 14 18:18:23.624616 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:23.624516    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/738b42b2-5340-4570-b696-d5de9540c6ea-maas-api-service-ca\") pod \"738b42b2-5340-4570-b696-d5de9540c6ea\" (UID: \"738b42b2-5340-4570-b696-d5de9540c6ea\") "
Jul 14 18:18:23.624879 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:23.624857    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738b42b2-5340-4570-b696-d5de9540c6ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "738b42b2-5340-4570-b696-d5de9540c6ea" (UID: "738b42b2-5340-4570-b696-d5de9540c6ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:18:23.725959 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:23.725916    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/738b42b2-5340-4570-b696-d5de9540c6ea-maas-api-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:24.368132 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:24.368094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc" event={"ID":"738b42b2-5340-4570-b696-d5de9540c6ea","Type":"ContainerDied","Data":"f30dce653f6ec73f19df90d5b306c8707f9b2bed7368d7d8025e19dfe52d27bf"}
Jul 14 18:18:24.368132 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:24.368121    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc"
Jul 14 18:18:24.368357 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:24.368189    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f30dce653f6ec73f19df90d5b306c8707f9b2bed7368d7d8025e19dfe52d27bf"
Jul 14 18:18:31.312469 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:31.312435    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"]
Jul 14 18:18:31.313089 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:31.312684    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl" podUID="c0479eb9-0695-49d3-8777-5e8803c65cac" containerName="maas-api" containerID="cri-o://26cb84000d171fae6c11c73ec3601732e8b5fbe1b881c229d68c40ee6a7a9104" gracePeriod=30
Jul 14 18:18:31.356251 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:31.356193    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:18:31.763771 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:31.763748    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:18:31.901586 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:31.901492    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0479eb9-0695-49d3-8777-5e8803c65cac-openshift-service-ca\") pod \"c0479eb9-0695-49d3-8777-5e8803c65cac\" (UID: \"c0479eb9-0695-49d3-8777-5e8803c65cac\") "
Jul 14 18:18:31.901586 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:31.901553    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0479eb9-0695-49d3-8777-5e8803c65cac-maas-api-tls\") pod \"c0479eb9-0695-49d3-8777-5e8803c65cac\" (UID: \"c0479eb9-0695-49d3-8777-5e8803c65cac\") "
Jul 14 18:18:31.901586 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:31.901587    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vdbn\" (UniqueName: \"kubernetes.io/projected/c0479eb9-0695-49d3-8777-5e8803c65cac-kube-api-access-5vdbn\") pod \"c0479eb9-0695-49d3-8777-5e8803c65cac\" (UID: \"c0479eb9-0695-49d3-8777-5e8803c65cac\") "
Jul 14 18:18:31.901907 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:31.901882    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0479eb9-0695-49d3-8777-5e8803c65cac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c0479eb9-0695-49d3-8777-5e8803c65cac" (UID: "c0479eb9-0695-49d3-8777-5e8803c65cac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:18:31.903812 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:31.903782    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0479eb9-0695-49d3-8777-5e8803c65cac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c0479eb9-0695-49d3-8777-5e8803c65cac" (UID: "c0479eb9-0695-49d3-8777-5e8803c65cac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:18:31.903812 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:31.903799    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0479eb9-0695-49d3-8777-5e8803c65cac-kube-api-access-5vdbn" (OuterVolumeSpecName: "kube-api-access-5vdbn") pod "c0479eb9-0695-49d3-8777-5e8803c65cac" (UID: "c0479eb9-0695-49d3-8777-5e8803c65cac"). InnerVolumeSpecName "kube-api-access-5vdbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:18:32.003148 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:32.003102    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0479eb9-0695-49d3-8777-5e8803c65cac-openshift-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:32.003148 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:32.003141    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0479eb9-0695-49d3-8777-5e8803c65cac-maas-api-tls\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:32.003148 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:32.003155    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vdbn\" (UniqueName: \"kubernetes.io/projected/c0479eb9-0695-49d3-8777-5e8803c65cac-kube-api-access-5vdbn\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:32.414736 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:32.414701    2578 generic.go:356] "Generic (PLEG): container finished" podID="c0479eb9-0695-49d3-8777-5e8803c65cac" containerID="26cb84000d171fae6c11c73ec3601732e8b5fbe1b881c229d68c40ee6a7a9104" exitCode=0
Jul 14 18:18:32.415189 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:32.414748    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"
Jul 14 18:18:32.415189 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:32.414750    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl" event={"ID":"c0479eb9-0695-49d3-8777-5e8803c65cac","Type":"ContainerDied","Data":"26cb84000d171fae6c11c73ec3601732e8b5fbe1b881c229d68c40ee6a7a9104"}
Jul 14 18:18:32.415189 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:32.414874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl" event={"ID":"c0479eb9-0695-49d3-8777-5e8803c65cac","Type":"ContainerDied","Data":"a8d1a2cec3a9d9b228acaae0d14296810b6285fd5cf7d73a300460e129a22398"}
Jul 14 18:18:32.415189 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:32.414906    2578 scope.go:117] "RemoveContainer" containerID="26cb84000d171fae6c11c73ec3601732e8b5fbe1b881c229d68c40ee6a7a9104"
Jul 14 18:18:32.425137 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:32.425116    2578 scope.go:117] "RemoveContainer" containerID="26cb84000d171fae6c11c73ec3601732e8b5fbe1b881c229d68c40ee6a7a9104"
Jul 14 18:18:32.425496 ip-10-0-135-231 kubenswrapper[2578]: E0714 18:18:32.425477    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26cb84000d171fae6c11c73ec3601732e8b5fbe1b881c229d68c40ee6a7a9104\": container with ID starting with 26cb84000d171fae6c11c73ec3601732e8b5fbe1b881c229d68c40ee6a7a9104 not found: ID does not exist" containerID="26cb84000d171fae6c11c73ec3601732e8b5fbe1b881c229d68c40ee6a7a9104"
Jul 14 18:18:32.425567 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:32.425505    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26cb84000d171fae6c11c73ec3601732e8b5fbe1b881c229d68c40ee6a7a9104"} err="failed to get container status \"26cb84000d171fae6c11c73ec3601732e8b5fbe1b881c229d68c40ee6a7a9104\": rpc error: code = NotFound desc = could not find container \"26cb84000d171fae6c11c73ec3601732e8b5fbe1b881c229d68c40ee6a7a9104\": container with ID starting with 26cb84000d171fae6c11c73ec3601732e8b5fbe1b881c229d68c40ee6a7a9104 not found: ID does not exist"
Jul 14 18:18:32.441522 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:32.441493    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"]
Jul 14 18:18:32.446011 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:32.445982    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-aa324e-5f774bccdd-mbvpl"]
Jul 14 18:18:33.783846 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:33.783809    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0479eb9-0695-49d3-8777-5e8803c65cac" path="/var/lib/kubelet/pods/c0479eb9-0695-49d3-8777-5e8803c65cac/volumes"
Jul 14 18:18:34.300454 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:34.300418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:18:35.839149 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:35.839116    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:18:42.294720 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:42.294687    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:18:42.394082 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:42.394050    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:18:46.557762 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:46.557670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:18:46.790178 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:46.790145    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:18:46.913785 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:46.913697    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs"]
Jul 14 18:18:46.914092 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:46.914079    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="738b42b2-5340-4570-b696-d5de9540c6ea" containerName="revoke-keys"
Jul 14 18:18:46.914150 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:46.914094    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="738b42b2-5340-4570-b696-d5de9540c6ea" containerName="revoke-keys"
Jul 14 18:18:46.914150 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:46.914107    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0479eb9-0695-49d3-8777-5e8803c65cac" containerName="maas-api"
Jul 14 18:18:46.914150 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:46.914112    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0479eb9-0695-49d3-8777-5e8803c65cac" containerName="maas-api"
Jul 14 18:18:46.914291 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:46.914175    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="738b42b2-5340-4570-b696-d5de9540c6ea" containerName="revoke-keys"
Jul 14 18:18:46.914291 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:46.914186    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0479eb9-0695-49d3-8777-5e8803c65cac" containerName="maas-api"
Jul 14 18:18:46.919113 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:46.919093    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs"
Jul 14 18:18:46.921730 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:46.921709    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pd7w9\""
Jul 14 18:18:46.928825 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:46.928795    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs"]
Jul 14 18:18:47.032754 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:47.032710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5528f053-4ad6-4294-a956-e3503eabe584-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs\" (UID: \"5528f053-4ad6-4294-a956-e3503eabe584\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs"
Jul 14 18:18:47.133523 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:47.133484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5528f053-4ad6-4294-a956-e3503eabe584-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs\" (UID: \"5528f053-4ad6-4294-a956-e3503eabe584\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs"
Jul 14 18:18:47.134171 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:47.134150    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5528f053-4ad6-4294-a956-e3503eabe584-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs\" (UID: \"5528f053-4ad6-4294-a956-e3503eabe584\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs"
Jul 14 18:18:47.232371 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:47.232336    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs"
Jul 14 18:18:47.375550 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:47.375518    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs"]
Jul 14 18:18:47.376646 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:18:47.376617    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5528f053_4ad6_4294_a956_e3503eabe584.slice/crio-fbcaa40b1e69d269df8446f58669e7daeffe1c0045a61665be5edb6723036d46 WatchSource:0}: Error finding container fbcaa40b1e69d269df8446f58669e7daeffe1c0045a61665be5edb6723036d46: Status 404 returned error can't find the container with id fbcaa40b1e69d269df8446f58669e7daeffe1c0045a61665be5edb6723036d46
Jul 14 18:18:47.507796 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:47.507692    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs" event={"ID":"5528f053-4ad6-4294-a956-e3503eabe584","Type":"ContainerStarted","Data":"fbcaa40b1e69d269df8446f58669e7daeffe1c0045a61665be5edb6723036d46"}
Jul 14 18:18:47.565267 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:47.565235    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:18:47.676491 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:47.676458    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:18:48.515312 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:48.515281    2578 generic.go:356] "Generic (PLEG): container finished" podID="5528f053-4ad6-4294-a956-e3503eabe584" containerID="07cc9e0102d1a02d01bfdf816ed372fd12ae575cbe5908e2e9b86d43db0bb676" exitCode=0
Jul 14 18:18:48.515508 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:48.515362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs" event={"ID":"5528f053-4ad6-4294-a956-e3503eabe584","Type":"ContainerDied","Data":"07cc9e0102d1a02d01bfdf816ed372fd12ae575cbe5908e2e9b86d43db0bb676"}
Jul 14 18:18:49.658987 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:49.658962    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs"
Jul 14 18:18:49.757046 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:49.757009    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5528f053-4ad6-4294-a956-e3503eabe584-maas-api-service-ca\") pod \"5528f053-4ad6-4294-a956-e3503eabe584\" (UID: \"5528f053-4ad6-4294-a956-e3503eabe584\") "
Jul 14 18:18:49.757419 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:49.757393    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5528f053-4ad6-4294-a956-e3503eabe584-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5528f053-4ad6-4294-a956-e3503eabe584" (UID: "5528f053-4ad6-4294-a956-e3503eabe584"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:18:49.858663 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:49.858615    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5528f053-4ad6-4294-a956-e3503eabe584-maas-api-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:50.528762 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:50.528722    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs" event={"ID":"5528f053-4ad6-4294-a956-e3503eabe584","Type":"ContainerDied","Data":"fbcaa40b1e69d269df8446f58669e7daeffe1c0045a61665be5edb6723036d46"}
Jul 14 18:18:50.528936 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:50.528771    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbcaa40b1e69d269df8446f58669e7daeffe1c0045a61665be5edb6723036d46"
Jul 14 18:18:50.528936 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:50.528729    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs"
Jul 14 18:18:57.423178 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:57.423146    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"]
Jul 14 18:18:57.423668 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:57.423394    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw" podUID="fdacffa5-490a-4070-9ce9-44e4baa297d1" containerName="maas-api" containerID="cri-o://928428acf2cd068770f51c851d58e87670f10d9dd2649fb08b56634c2ef28473" gracePeriod=30
Jul 14 18:18:57.465590 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:57.465561    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:18:57.573071 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:57.573037    2578 generic.go:356] "Generic (PLEG): container finished" podID="fdacffa5-490a-4070-9ce9-44e4baa297d1" containerID="928428acf2cd068770f51c851d58e87670f10d9dd2649fb08b56634c2ef28473" exitCode=0
Jul 14 18:18:57.573271 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:57.573115    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw" event={"ID":"fdacffa5-490a-4070-9ce9-44e4baa297d1","Type":"ContainerDied","Data":"928428acf2cd068770f51c851d58e87670f10d9dd2649fb08b56634c2ef28473"}
Jul 14 18:18:57.673781 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:57.673715    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:18:57.735243 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:57.735187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdacffa5-490a-4070-9ce9-44e4baa297d1-maas-api-tls\") pod \"fdacffa5-490a-4070-9ce9-44e4baa297d1\" (UID: \"fdacffa5-490a-4070-9ce9-44e4baa297d1\") "
Jul 14 18:18:57.735404 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:57.735257    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9455\" (UniqueName: \"kubernetes.io/projected/fdacffa5-490a-4070-9ce9-44e4baa297d1-kube-api-access-k9455\") pod \"fdacffa5-490a-4070-9ce9-44e4baa297d1\" (UID: \"fdacffa5-490a-4070-9ce9-44e4baa297d1\") "
Jul 14 18:18:57.735404 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:57.735396    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdacffa5-490a-4070-9ce9-44e4baa297d1-openshift-service-ca\") pod \"fdacffa5-490a-4070-9ce9-44e4baa297d1\" (UID: \"fdacffa5-490a-4070-9ce9-44e4baa297d1\") "
Jul 14 18:18:57.735749 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:57.735729    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdacffa5-490a-4070-9ce9-44e4baa297d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fdacffa5-490a-4070-9ce9-44e4baa297d1" (UID: "fdacffa5-490a-4070-9ce9-44e4baa297d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:18:57.737416 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:57.737387    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdacffa5-490a-4070-9ce9-44e4baa297d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fdacffa5-490a-4070-9ce9-44e4baa297d1" (UID: "fdacffa5-490a-4070-9ce9-44e4baa297d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:18:57.737526 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:57.737481    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdacffa5-490a-4070-9ce9-44e4baa297d1-kube-api-access-k9455" (OuterVolumeSpecName: "kube-api-access-k9455") pod "fdacffa5-490a-4070-9ce9-44e4baa297d1" (UID: "fdacffa5-490a-4070-9ce9-44e4baa297d1"). InnerVolumeSpecName "kube-api-access-k9455". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:18:57.836512 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:57.836480    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdacffa5-490a-4070-9ce9-44e4baa297d1-openshift-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:57.836512 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:57.836506    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdacffa5-490a-4070-9ce9-44e4baa297d1-maas-api-tls\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:57.836512 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:57.836515    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9455\" (UniqueName: \"kubernetes.io/projected/fdacffa5-490a-4070-9ce9-44e4baa297d1-kube-api-access-k9455\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jul 14 18:18:58.580094 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:58.580064    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"
Jul 14 18:18:58.580094 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:58.580080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw" event={"ID":"fdacffa5-490a-4070-9ce9-44e4baa297d1","Type":"ContainerDied","Data":"09a51811e3a9047c34c226ead609170918999a37d682bcb3654c8ae7334a04db"}
Jul 14 18:18:58.580650 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:58.580136    2578 scope.go:117] "RemoveContainer" containerID="928428acf2cd068770f51c851d58e87670f10d9dd2649fb08b56634c2ef28473"
Jul 14 18:18:58.602011 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:58.601984    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"]
Jul 14 18:18:58.604546 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:58.604526    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-877bd7-7d8bbddb94-z75gw"]
Jul 14 18:18:59.784131 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:18:59.784094    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdacffa5-490a-4070-9ce9-44e4baa297d1" path="/var/lib/kubelet/pods/fdacffa5-490a-4070-9ce9-44e4baa297d1/volumes"
Jul 14 18:19:00.205612 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:00.205571    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x7kzr"]
Jul 14 18:19:02.070084 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:02.070044    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:19:08.957190 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:08.957156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:19:09.062799 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:09.062769    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:19:12.814838 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:12.814799    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:19:13.053232 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:13.053178    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:19:13.407732 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:13.407699    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85955854f9-7kr6n_9dd33441-61e5-4b5f-b2ff-875644fffc2c/authorino/0.log"
Jul 14 18:19:18.292663 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:18.292629    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kr7sr_ef5a476a-8d1d-446f-911d-470497232b26/manager/0.log"
Jul 14 18:19:18.409127 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:18.409093    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bwbg7_9b13f954-fcce-4664-bc31-d1b20cfc25f9/manager/0.log"
Jul 14 18:19:18.686372 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:18.686345    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-k497s_d8075eca-f4e2-4595-ab77-3da2f78e0506/server/0.log"
Jul 14 18:19:18.805389 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:18.805360    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-d9mq2_c29c7995-8119-4a2d-b7bb-0b233ac8e1c8/manager/0.log"
Jul 14 18:19:19.061054 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:19.060969    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7558c48c44-hlgbh_e41e0738-0f4b-4e94-bf50-a2482c47a957/copy-manifests/0.log"
Jul 14 18:19:19.070219 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:19.070174    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7558c48c44-hlgbh_e41e0738-0f4b-4e94-bf50-a2482c47a957/manager/0.log"
Jul 14 18:19:19.772616 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:19.772590    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734200-swcb7_527c26cc-77b2-4188-8a57-be5c367992a6/cleanup/0.log"
Jul 14 18:19:19.989356 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:19.989328    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-09df0cc0-5qmn5_f19c51af-d6a1-4a60-a1b0-c0db9a84c501/revoke-keys/0.log"
Jul 14 18:19:20.102816 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:20.102744    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-439ad9d2-s6thz_932277ae-a14f-41b7-93b1-07fddff76e5b/revoke-keys/0.log"
Jul 14 18:19:20.216699 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:20.216670    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-5a43f9-jhbr5_b0dfb536-7f3a-4319-bbd3-ad40d63d7e30/revoke-keys/0.log"
Jul 14 18:19:20.340581 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:20.340553    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-e9d840-rsv6n_2e86dc48-8aa8-4a69-b21f-d1601cc8d31b/revoke-keys/0.log"
Jul 14 18:19:20.461262 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:20.461232    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-68125063-kznjt_f7013419-68fa-4ef1-b415-0c501e86a769/revoke-keys/0.log"
Jul 14 18:19:20.680420 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:20.680385    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-9c28f2-sbtfg_a512ffcd-7e14-492e-8aa2-718e31db270b/revoke-keys/0.log"
Jul 14 18:19:20.794080 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:20.793999    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-b28b559b-s8pkb_99971710-4cf0-4461-810c-bf875fbfcafb/revoke-keys/0.log"
Jul 14 18:19:20.908944 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:20.908918    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-8cb2afcb-q85bd_4a3f0ea4-3f24-460e-8478-daf0fe14b169/revoke-keys/0.log"
Jul 14 18:19:21.020238 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:21.020192    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-ef19c3e1-rwks2_e6780df0-ec66-463c-a21c-5ad5360b5b68/revoke-keys/0.log"
Jul 14 18:19:21.136433 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:21.136361    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-aa324e-wjcpc_738b42b2-5340-4570-b696-d5de9540c6ea/revoke-keys/0.log"
Jul 14 18:19:21.244722 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:21.244693    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-877bd7-8wpgs_5528f053-4ad6-4294-a956-e3503eabe584/revoke-keys/0.log"
Jul 14 18:19:22.134032 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:22.134000    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5_2b693041-1102-4307-ade2-6afd89206f18/util/0.log"
Jul 14 18:19:22.143786 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:22.143765    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5_2b693041-1102-4307-ade2-6afd89206f18/pull/0.log"
Jul 14 18:19:22.151830 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:22.151810    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5_2b693041-1102-4307-ade2-6afd89206f18/extract/0.log"
Jul 14 18:19:22.270532 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:22.270504    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp_d7df75c1-34d5-4e45-8c2f-d30cc4d29d48/util/0.log"
Jul 14 18:19:22.279396 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:22.279369    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp_d7df75c1-34d5-4e45-8c2f-d30cc4d29d48/pull/0.log"
Jul 14 18:19:22.290445 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:22.290425    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp_d7df75c1-34d5-4e45-8c2f-d30cc4d29d48/extract/0.log"
Jul 14 18:19:22.406448 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:22.406365    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp_24621efc-cea7-4df7-a5af-e02a4ab5a8d2/util/0.log"
Jul 14 18:19:22.415583 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:22.415564    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp_24621efc-cea7-4df7-a5af-e02a4ab5a8d2/pull/0.log"
Jul 14 18:19:22.423076 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:22.423058    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp_24621efc-cea7-4df7-a5af-e02a4ab5a8d2/extract/0.log"
Jul 14 18:19:22.544863 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:22.544838    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd_4be48f74-1675-42e1-a07f-6207e22d2a18/util/0.log"
Jul 14 18:19:22.552759 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:22.552730    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd_4be48f74-1675-42e1-a07f-6207e22d2a18/pull/0.log"
Jul 14 18:19:22.560804 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:22.560782    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd_4be48f74-1675-42e1-a07f-6207e22d2a18/extract/0.log"
Jul 14 18:19:22.676843 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:22.676814    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85955854f9-7kr6n_9dd33441-61e5-4b5f-b2ff-875644fffc2c/authorino/0.log"
Jul 14 18:19:22.798138 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:22.798109    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-jpqgl_6fe9e15d-bb8e-4c0a-a39f-3f57397e01bd/manager/0.log"
Jul 14 18:19:22.912176 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:22.912150    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ffz54_f625fea1-c044-4e41-9652-2311ff48ac52/manager/0.log"
Jul 14 18:19:23.028480 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:23.028405    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rhrps_1803e1bd-4131-42bb-844a-4ef466c6ddcd/kuadrant-console-plugin/0.log"
Jul 14 18:19:23.150531 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:23.150505    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-s425v_90bc43ea-bf7c-4ffb-88cc-7077753a16b1/registry-server/0.log"
Jul 14 18:19:23.389764 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:23.389684    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-x7kzr_ca5f0f7f-bb5f-484b-8aa1-9754083cb27a/limitador/0.log"
Jul 14 18:19:23.507477 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:23.507448    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hfcjm_c3c1b6d8-3a4d-4205-8df9-104feee899c4/manager/0.log"
Jul 14 18:19:24.213087 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:24.213058    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fd75767d5-j5vhv_cd61ccf1-5789-425f-b22b-77db95bd220b/kube-auth-proxy/0.log"
Jul 14 18:19:25.157752 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:25.157711    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-k78cm_a964ccaf-1a5b-49b2-b566-7c755e3dfc31/main/0.log"
Jul 14 18:19:25.418081 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:25.418000    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zzkp_fed6a600-9415-4bdf-98a5-3725e69ffe93/main/0.log"
Jul 14 18:19:25.658037 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:25.658004    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-m775m_0a5b06c5-973f-40bd-8782-ce489f59e66d/main/0.log"
Jul 14 18:19:30.063550 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.063516    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g2pvf/must-gather-9jgqf"]
Jul 14 18:19:30.064001 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.063922    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5528f053-4ad6-4294-a956-e3503eabe584" containerName="revoke-keys"
Jul 14 18:19:30.064001 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.063933    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5528f053-4ad6-4294-a956-e3503eabe584" containerName="revoke-keys"
Jul 14 18:19:30.064001 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.063946    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdacffa5-490a-4070-9ce9-44e4baa297d1" containerName="maas-api"
Jul 14 18:19:30.064001 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.063951    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdacffa5-490a-4070-9ce9-44e4baa297d1" containerName="maas-api"
Jul 14 18:19:30.064136 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.064007    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdacffa5-490a-4070-9ce9-44e4baa297d1" containerName="maas-api"
Jul 14 18:19:30.064136 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.064015    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5528f053-4ad6-4294-a956-e3503eabe584" containerName="revoke-keys"
Jul 14 18:19:30.068738 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.068716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g2pvf/must-gather-9jgqf"
Jul 14 18:19:30.071089 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.071071    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g2pvf\"/\"kube-root-ca.crt\""
Jul 14 18:19:30.071735 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.071722    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g2pvf\"/\"default-dockercfg-4w9h4\""
Jul 14 18:19:30.071792 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.071730    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g2pvf\"/\"openshift-service-ca.crt\""
Jul 14 18:19:30.082371 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.082345    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g2pvf/must-gather-9jgqf"]
Jul 14 18:19:30.101255 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.101226    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:19:30.230663 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.230631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scfz9\" (UniqueName: \"kubernetes.io/projected/14504fb9-fa8f-4d75-bb73-7ec3b1dd65fd-kube-api-access-scfz9\") pod \"must-gather-9jgqf\" (UID: \"14504fb9-fa8f-4d75-bb73-7ec3b1dd65fd\") " pod="openshift-must-gather-g2pvf/must-gather-9jgqf"
Jul 14 18:19:30.230819 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.230670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/14504fb9-fa8f-4d75-bb73-7ec3b1dd65fd-must-gather-output\") pod \"must-gather-9jgqf\" (UID: \"14504fb9-fa8f-4d75-bb73-7ec3b1dd65fd\") " pod="openshift-must-gather-g2pvf/must-gather-9jgqf"
Jul 14 18:19:30.250647 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.250606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:19:30.332238 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.332121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scfz9\" (UniqueName: \"kubernetes.io/projected/14504fb9-fa8f-4d75-bb73-7ec3b1dd65fd-kube-api-access-scfz9\") pod \"must-gather-9jgqf\" (UID: \"14504fb9-fa8f-4d75-bb73-7ec3b1dd65fd\") " pod="openshift-must-gather-g2pvf/must-gather-9jgqf"
Jul 14 18:19:30.332238 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.332163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/14504fb9-fa8f-4d75-bb73-7ec3b1dd65fd-must-gather-output\") pod \"must-gather-9jgqf\" (UID: \"14504fb9-fa8f-4d75-bb73-7ec3b1dd65fd\") " pod="openshift-must-gather-g2pvf/must-gather-9jgqf"
Jul 14 18:19:30.332519 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.332500    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/14504fb9-fa8f-4d75-bb73-7ec3b1dd65fd-must-gather-output\") pod \"must-gather-9jgqf\" (UID: \"14504fb9-fa8f-4d75-bb73-7ec3b1dd65fd\") " pod="openshift-must-gather-g2pvf/must-gather-9jgqf"
Jul 14 18:19:30.340563 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.340543    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scfz9\" (UniqueName: \"kubernetes.io/projected/14504fb9-fa8f-4d75-bb73-7ec3b1dd65fd-kube-api-access-scfz9\") pod \"must-gather-9jgqf\" (UID: \"14504fb9-fa8f-4d75-bb73-7ec3b1dd65fd\") " pod="openshift-must-gather-g2pvf/must-gather-9jgqf"
Jul 14 18:19:30.378822 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.378793    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g2pvf/must-gather-9jgqf"
Jul 14 18:19:30.538424 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.538397    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g2pvf/must-gather-9jgqf"]
Jul 14 18:19:30.540228 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:19:30.540183    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14504fb9_fa8f_4d75_bb73_7ec3b1dd65fd.slice/crio-8783aa8a36ed5cdfef9be2783079e0cb10a9b226fccbd094c2c3defe38c49e1e WatchSource:0}: Error finding container 8783aa8a36ed5cdfef9be2783079e0cb10a9b226fccbd094c2c3defe38c49e1e: Status 404 returned error can't find the container with id 8783aa8a36ed5cdfef9be2783079e0cb10a9b226fccbd094c2c3defe38c49e1e
Jul 14 18:19:30.774188 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:30.774156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g2pvf/must-gather-9jgqf" event={"ID":"14504fb9-fa8f-4d75-bb73-7ec3b1dd65fd","Type":"ContainerStarted","Data":"8783aa8a36ed5cdfef9be2783079e0cb10a9b226fccbd094c2c3defe38c49e1e"}
Jul 14 18:19:34.441456 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:34.441352    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log" limit=1024
Jul 14 18:19:34.802650 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:34.802541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g2pvf/must-gather-9jgqf" event={"ID":"14504fb9-fa8f-4d75-bb73-7ec3b1dd65fd","Type":"ContainerStarted","Data":"1b4633e8d8787dc015af442b69d72c388e90a77944bdd40a765e329f6fe57fc5"}
Jul 14 18:19:34.802650 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:34.802590    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g2pvf/must-gather-9jgqf" event={"ID":"14504fb9-fa8f-4d75-bb73-7ec3b1dd65fd","Type":"ContainerStarted","Data":"ce38b38a1fa99d17872aa1cd027a60b0bc25e44fa50f3cb82213d6ced2e3696e"}
Jul 14 18:19:34.821172 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:34.821114    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g2pvf/must-gather-9jgqf" podStartSLOduration=1.482280096 podStartE2EDuration="4.821097669s" totalImagesPullingTime="3.338817573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:19:34.820417096 +0000 UTC m=+2831.536810788" watchObservedRunningTime="2026-07-14 18:19:34.821097669 +0000 UTC m=+2831.537491361"
Jul 14 18:19:35.786334 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:35.786309    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fk7h5_fb86cfa4-c36d-43f2-b8ac-3ceb635b59f7/global-pull-secret-syncer/0.log"
Jul 14 18:19:36.039423 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:36.039337    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fdndd_5afed38b-3690-4505-9749-3d026e206a4f/konnectivity-agent/0.log"
Jul 14 18:19:36.167730 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:36.167697    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-231.ec2.internal_e8a89385d90f63ff945cda7868eed5df/haproxy/0.log"
Jul 14 18:19:40.536331 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:40.536304    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5_2b693041-1102-4307-ade2-6afd89206f18/extract/0.log"
Jul 14 18:19:40.571022 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:40.570995    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5_2b693041-1102-4307-ade2-6afd89206f18/util/0.log"
Jul 14 18:19:40.599794 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:40.599743    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h92p5_2b693041-1102-4307-ade2-6afd89206f18/pull/0.log"
Jul 14 18:19:40.639641 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:40.639606    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp_d7df75c1-34d5-4e45-8c2f-d30cc4d29d48/extract/0.log"
Jul 14 18:19:40.677599 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:40.677564    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp_d7df75c1-34d5-4e45-8c2f-d30cc4d29d48/util/0.log"
Jul 14 18:19:40.708875 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:40.708849    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04dlxp_d7df75c1-34d5-4e45-8c2f-d30cc4d29d48/pull/0.log"
Jul 14 18:19:40.742063 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:40.742032    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp_24621efc-cea7-4df7-a5af-e02a4ab5a8d2/extract/0.log"
Jul 14 18:19:40.772358 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:40.772329    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp_24621efc-cea7-4df7-a5af-e02a4ab5a8d2/util/0.log"
Jul 14 18:19:40.799941 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:40.799861    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq7pp_24621efc-cea7-4df7-a5af-e02a4ab5a8d2/pull/0.log"
Jul 14 18:19:40.834415 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:40.834388    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd_4be48f74-1675-42e1-a07f-6207e22d2a18/extract/0.log"
Jul 14 18:19:40.858189 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:40.858159    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd_4be48f74-1675-42e1-a07f-6207e22d2a18/util/0.log"
Jul 14 18:19:40.882668 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:40.882633    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dnqmd_4be48f74-1675-42e1-a07f-6207e22d2a18/pull/0.log"
Jul 14 18:19:41.312228 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:41.311957    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85955854f9-7kr6n_9dd33441-61e5-4b5f-b2ff-875644fffc2c/authorino/0.log"
Jul 14 18:19:41.349226 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:41.348921    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-jpqgl_6fe9e15d-bb8e-4c0a-a39f-3f57397e01bd/manager/0.log"
Jul 14 18:19:41.380696 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:41.380663    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ffz54_f625fea1-c044-4e41-9652-2311ff48ac52/manager/0.log"
Jul 14 18:19:41.407862 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:41.407838    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rhrps_1803e1bd-4131-42bb-844a-4ef466c6ddcd/kuadrant-console-plugin/0.log"
Jul 14 18:19:41.457368 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:41.457335    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-s425v_90bc43ea-bf7c-4ffb-88cc-7077753a16b1/registry-server/0.log"
Jul 14 18:19:41.629111 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:41.629025    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-x7kzr_ca5f0f7f-bb5f-484b-8aa1-9754083cb27a/limitador/0.log"
Jul 14 18:19:41.713196 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:41.713166    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hfcjm_c3c1b6d8-3a4d-4205-8df9-104feee899c4/manager/0.log"
Jul 14 18:19:43.822141 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:43.822082    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-gsw6x_d483aff6-f6b3-4ab6-836e-3817bfa065a6/kube-state-metrics/0.log"
Jul 14 18:19:43.860654 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:43.860625    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-gsw6x_d483aff6-f6b3-4ab6-836e-3817bfa065a6/kube-rbac-proxy-main/0.log"
Jul 14 18:19:43.890820 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:43.890745    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-gsw6x_d483aff6-f6b3-4ab6-836e-3817bfa065a6/kube-rbac-proxy-self/0.log"
Jul 14 18:19:43.988112 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:43.988087    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-hc5dt_c0a12187-5a00-4ab2-a043-37511faad495/monitoring-plugin/0.log"
Jul 14 18:19:44.028804 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:44.028766    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5cr5d_5f3e5c41-2310-4cb6-b57b-2f3407c09c69/node-exporter/0.log"
Jul 14 18:19:44.053894 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:44.053866    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5cr5d_5f3e5c41-2310-4cb6-b57b-2f3407c09c69/kube-rbac-proxy/0.log"
Jul 14 18:19:44.080108 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:44.080032    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5cr5d_5f3e5c41-2310-4cb6-b57b-2f3407c09c69/init-textfile/0.log"
Jul 14 18:19:44.383515 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:44.383434    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-nd8j5_9ee90214-e9d3-4400-8170-7a6fd3a39652/kube-rbac-proxy-main/0.log"
Jul 14 18:19:44.409536 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:44.409503    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-nd8j5_9ee90214-e9d3-4400-8170-7a6fd3a39652/kube-rbac-proxy-self/0.log"
Jul 14 18:19:44.436367 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:44.436335    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-nd8j5_9ee90214-e9d3-4400-8170-7a6fd3a39652/openshift-state-metrics/0.log"
Jul 14 18:19:44.693838 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:44.693788    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-5x6wl_cffc77fc-29b6-4126-9b0b-3ed4fc964ee2/prometheus-operator/0.log"
Jul 14 18:19:44.718324 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:44.718270    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-5x6wl_cffc77fc-29b6-4126-9b0b-3ed4fc964ee2/kube-rbac-proxy/0.log"
Jul 14 18:19:45.553454 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.553422    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"]
Jul 14 18:19:45.561292 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.561260    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.569304 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.569277    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"]
Jul 14 18:19:45.695063 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.695033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/485f6b08-ad17-44c8-86ad-13be403acd60-sys\") pod \"perf-node-gather-daemonset-nsh6b\" (UID: \"485f6b08-ad17-44c8-86ad-13be403acd60\") " pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.695063 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.695067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdwm\" (UniqueName: \"kubernetes.io/projected/485f6b08-ad17-44c8-86ad-13be403acd60-kube-api-access-5jdwm\") pod \"perf-node-gather-daemonset-nsh6b\" (UID: \"485f6b08-ad17-44c8-86ad-13be403acd60\") " pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.695300 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.695114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/485f6b08-ad17-44c8-86ad-13be403acd60-podres\") pod \"perf-node-gather-daemonset-nsh6b\" (UID: \"485f6b08-ad17-44c8-86ad-13be403acd60\") " pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.695300 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.695193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/485f6b08-ad17-44c8-86ad-13be403acd60-lib-modules\") pod \"perf-node-gather-daemonset-nsh6b\" (UID: \"485f6b08-ad17-44c8-86ad-13be403acd60\") " pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.695378 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.695305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/485f6b08-ad17-44c8-86ad-13be403acd60-proc\") pod \"perf-node-gather-daemonset-nsh6b\" (UID: \"485f6b08-ad17-44c8-86ad-13be403acd60\") " pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.796494 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.796462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/485f6b08-ad17-44c8-86ad-13be403acd60-sys\") pod \"perf-node-gather-daemonset-nsh6b\" (UID: \"485f6b08-ad17-44c8-86ad-13be403acd60\") " pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.796494 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.796497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdwm\" (UniqueName: \"kubernetes.io/projected/485f6b08-ad17-44c8-86ad-13be403acd60-kube-api-access-5jdwm\") pod \"perf-node-gather-daemonset-nsh6b\" (UID: \"485f6b08-ad17-44c8-86ad-13be403acd60\") " pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.796754 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.796545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/485f6b08-ad17-44c8-86ad-13be403acd60-podres\") pod \"perf-node-gather-daemonset-nsh6b\" (UID: \"485f6b08-ad17-44c8-86ad-13be403acd60\") " pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.796754 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.796573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/485f6b08-ad17-44c8-86ad-13be403acd60-lib-modules\") pod \"perf-node-gather-daemonset-nsh6b\" (UID: \"485f6b08-ad17-44c8-86ad-13be403acd60\") " pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.796754 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.796603    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/485f6b08-ad17-44c8-86ad-13be403acd60-sys\") pod \"perf-node-gather-daemonset-nsh6b\" (UID: \"485f6b08-ad17-44c8-86ad-13be403acd60\") " pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.796754 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.796618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/485f6b08-ad17-44c8-86ad-13be403acd60-proc\") pod \"perf-node-gather-daemonset-nsh6b\" (UID: \"485f6b08-ad17-44c8-86ad-13be403acd60\") " pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.796754 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.796674    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/485f6b08-ad17-44c8-86ad-13be403acd60-proc\") pod \"perf-node-gather-daemonset-nsh6b\" (UID: \"485f6b08-ad17-44c8-86ad-13be403acd60\") " pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.796754 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.796724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/485f6b08-ad17-44c8-86ad-13be403acd60-lib-modules\") pod \"perf-node-gather-daemonset-nsh6b\" (UID: \"485f6b08-ad17-44c8-86ad-13be403acd60\") " pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.796754 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.796736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/485f6b08-ad17-44c8-86ad-13be403acd60-podres\") pod \"perf-node-gather-daemonset-nsh6b\" (UID: \"485f6b08-ad17-44c8-86ad-13be403acd60\") " pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.806687 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.806607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdwm\" (UniqueName: \"kubernetes.io/projected/485f6b08-ad17-44c8-86ad-13be403acd60-kube-api-access-5jdwm\") pod \"perf-node-gather-daemonset-nsh6b\" (UID: \"485f6b08-ad17-44c8-86ad-13be403acd60\") " pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:45.877354 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:45.877317    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:46.079515 ip-10-0-135-231 kubenswrapper[2578]: W0714 18:19:46.079473    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod485f6b08_ad17_44c8_86ad_13be403acd60.slice/crio-d6ea306c54c9f378100119bcfdbf598d2826b8065e025b88ce1af9e81554a285 WatchSource:0}: Error finding container d6ea306c54c9f378100119bcfdbf598d2826b8065e025b88ce1af9e81554a285: Status 404 returned error can't find the container with id d6ea306c54c9f378100119bcfdbf598d2826b8065e025b88ce1af9e81554a285
Jul 14 18:19:46.079971 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:46.079949    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"]
Jul 14 18:19:46.922246 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:46.922177    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b" event={"ID":"485f6b08-ad17-44c8-86ad-13be403acd60","Type":"ContainerStarted","Data":"01ebbbd8435d74dc080671b91e6993c6a5c16a92c2921cef1b2190f24450955e"}
Jul 14 18:19:46.922667 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:46.922278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b" event={"ID":"485f6b08-ad17-44c8-86ad-13be403acd60","Type":"ContainerStarted","Data":"d6ea306c54c9f378100119bcfdbf598d2826b8065e025b88ce1af9e81554a285"}
Jul 14 18:19:46.922667 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:46.922409    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:46.941860 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:46.941808    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b" podStartSLOduration=1.941792679 podStartE2EDuration="1.941792679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:19:46.939752786 +0000 UTC m=+2843.656146480" watchObservedRunningTime="2026-07-14 18:19:46.941792679 +0000 UTC m=+2843.658186371"
Jul 14 18:19:47.356792 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:47.356716    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-bb6466c6d-bfh8h_f2a5b360-82f8-4b9c-8ce2-1b1516b82f2d/console/0.log"
Jul 14 18:19:49.178829 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:49.178805    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f76mp_89dbd5c1-c9c4-4182-83e4-b1c086db1654/dns/0.log"
Jul 14 18:19:49.209659 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:49.209633    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f76mp_89dbd5c1-c9c4-4182-83e4-b1c086db1654/kube-rbac-proxy/0.log"
Jul 14 18:19:49.357992 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:49.357962    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-786wn_52e52a4c-3f78-40f6-8d9e-963ff65c0f86/dns-node-resolver/0.log"
Jul 14 18:19:50.047932 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:50.047899    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-f6b6674b9-4lr8w_a6f3fcfd-03da-490d-aef8-06ef2c6875bb/registry/0.log"
Jul 14 18:19:50.075281 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:50.075244    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-554b4_6f85b40b-bca9-4452-ac81-5786ca39e3e4/node-ca/0.log"
Jul 14 18:19:51.417790 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:51.417764    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fd75767d5-j5vhv_cd61ccf1-5789-425f-b22b-77db95bd220b/kube-auth-proxy/0.log"
Jul 14 18:19:52.327648 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:52.327620    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tcqk5_2317c923-3e9d-4bee-b3fe-760df78d2176/serve-healthcheck-canary/0.log"
Jul 14 18:19:52.944258 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:52.944224    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g2pvf/perf-node-gather-daemonset-nsh6b"
Jul 14 18:19:52.993596 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:52.993565    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s48hg_4186b2ec-a392-40d8-b06d-8be76fa35bbe/kube-rbac-proxy/0.log"
Jul 14 18:19:53.024012 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:53.023986    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s48hg_4186b2ec-a392-40d8-b06d-8be76fa35bbe/exporter/0.log"
Jul 14 18:19:53.052077 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:53.052050    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s48hg_4186b2ec-a392-40d8-b06d-8be76fa35bbe/extractor/0.log"
Jul 14 18:19:55.716853 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:55.716820    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kr7sr_ef5a476a-8d1d-446f-911d-470497232b26/manager/0.log"
Jul 14 18:19:55.906093 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:55.906052    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bwbg7_9b13f954-fcce-4664-bc31-d1b20cfc25f9/manager/0.log"
Jul 14 18:19:56.110745 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:56.110641    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-k497s_d8075eca-f4e2-4595-ab77-3da2f78e0506/server/0.log"
Jul 14 18:19:56.241732 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:56.241701    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-d9mq2_c29c7995-8119-4a2d-b7bb-0b233ac8e1c8/manager/0.log"
Jul 14 18:19:56.371255 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:56.371129    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7558c48c44-hlgbh_e41e0738-0f4b-4e94-bf50-a2482c47a957/manager/0.log"
Jul 14 18:19:56.411151 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:56.411126    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7558c48c44-hlgbh_e41e0738-0f4b-4e94-bf50-a2482c47a957/copy-manifests/0.log"
Jul 14 18:19:56.539687 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:56.539662    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fd75767d5-j5vhv_cd61ccf1-5789-425f-b22b-77db95bd220b/kube-auth-proxy/0.log"
Jul 14 18:19:57.957481 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:57.957443    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fcdmf_f45c09af-344b-4666-a2d3-f8da53eea0bb/openshift-lws-operator/0.log"
Jul 14 18:19:58.070704 ip-10-0-135-231 kubenswrapper[2578]: I0714 18:19:58.070665    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fd75767d5-j5vhv_cd61ccf1-5789-425f-b22b-77db95bd220b/kube-auth-proxy/0.log"