Sep 15 18:50:36.298735 ip-10-0-129-227 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 18:50:36.673486 ip-10-0-129-227 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 18:50:36.673486 ip-10-0-129-227 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 18:50:36.673486 ip-10-0-129-227 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 18:50:36.673486 ip-10-0-129-227 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 18:50:36.673486 ip-10-0-129-227 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 18:50:36.676068 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.675977    2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 18:50:36.681623 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681605    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:36.681623 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681623    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681627    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681630    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681641    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681644    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681647    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681650    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681653    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681655    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681658    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681661    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681663    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681666    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681669    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681673    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681675    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681678    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681682    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681686    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:36.681777 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681689    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681693    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681696    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681699    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681702    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681705    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681708    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681710    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681713    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681717    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681721    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681724    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681727    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681729    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681732    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681735    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681738    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681742    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681744    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681747    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:36.682254 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681750    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681752    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681755    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681758    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681760    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681763    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681765    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681768    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681770    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681773    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681775    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681778    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681781    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681783    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681786    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681789    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681791    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681794    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681797    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681799    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:36.682752 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681802    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681804    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681808    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681810    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681813    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681816    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681818    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681823    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681826    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681829    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681832    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681835    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681838    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681840    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681843    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681846    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681848    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681852    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681855    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681858    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:36.683259 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681863    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681865    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681868    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681871    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681874    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681876    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681879    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681881    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681884    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681887    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681891    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681894    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681897    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681900    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681904    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681907    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681910    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681912    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.681915    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:36.683780 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682631    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682638    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682642    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682645    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682649    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682652    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682655    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682658    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682660    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682663    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682666    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682670    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682674    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682678    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682681    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682684    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682687    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682690    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682692    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682695    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:36.684238 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682698    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682700    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682703    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682705    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682708    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682710    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682713    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682715    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682718    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682720    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682723    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682726    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682729    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682732    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682734    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682737    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682740    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682743    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682745    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682748    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:36.684794 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682751    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682754    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682756    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682759    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682761    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682764    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682767    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682769    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682772    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682774    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682778    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682780    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682783    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682785    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682788    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682791    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682793    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682795    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682798    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682801    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:36.685318 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682804    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682807    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682809    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682813    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682815    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682818    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682821    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682823    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682826    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682829    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682832    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682835    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682837    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682840    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682842    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682845    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682847    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682850    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682853    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682855    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:36.686108 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682857    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682862    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682866    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682868    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682871    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682873    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682876    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682878    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682882    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682886    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682889    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682892    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682896    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682899    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682902    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682905    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682908    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682910    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.682913    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:36.686747 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683304    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683309    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683312    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683315    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683317    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683320    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683323    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683326    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683328    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683331    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683333    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683336    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683339    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683341    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683344    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683349    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683352    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683356    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683358    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683361    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:36.687252 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683363    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683367    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683370    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683372    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683375    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683381    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683383    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683387    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683390    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683392    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683395    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683398    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683401    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683403    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683406    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683408    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683411    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683414    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683417    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683420    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:36.687981 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683422    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683425    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683428    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683430    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683433    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683456    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683459    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683461    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683473    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683477    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683481    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683484    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683487    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683490    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683492    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683495    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683498    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683503    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683506    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683509    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:36.688537 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683511    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683514    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683517    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683520    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683523    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683525    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683528    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683530    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683535    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683538    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683541    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683544    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683546    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683549    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683552    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683556    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683560    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683563    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683566    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:36.689041 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683569    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683572    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683575    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683578    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683581    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683584    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683587    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683590    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683592    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683595    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683599    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683602    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683604    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683607    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683609    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683612    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683614    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683617    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683619    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.683622    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:36.689652 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683701    2550 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683737    2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683745    2550 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683750    2550 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683754    2550 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683758    2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683762    2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683766    2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683769    2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683773    2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683776    2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683780    2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683783    2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683786    2550 flags.go:64] FLAG: --cgroup-root=""
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683790    2550 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683793    2550 flags.go:64] FLAG: --client-ca-file=""
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683796    2550 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683799    2550 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683804    2550 flags.go:64] FLAG: --cluster-domain=""
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683807    2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683810    2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683813    2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683817    2550 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683821    2550 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 18:50:36.690157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683824    2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683828    2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683832    2550 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683835    2550 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683838    2550 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683841    2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683844    2550 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683847    2550 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683851    2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683855    2550 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683858    2550 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683861    2550 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683865    2550 flags.go:64] FLAG: --enable-server="true"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683868    2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683873    2550 flags.go:64] FLAG: --event-burst="100"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683876    2550 flags.go:64] FLAG: --event-qps="50"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683879    2550 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683882    2550 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683885    2550 flags.go:64] FLAG: --eviction-hard=""
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683889    2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683891    2550 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683894    2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683897    2550 flags.go:64] FLAG: --eviction-soft=""
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683900    2550 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683903    2550 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 18:50:36.690751 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683906    2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683910    2550 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683913    2550 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683916    2550 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683919    2550 flags.go:64] FLAG: --feature-gates=""
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683923    2550 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683926    2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683929    2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683932    2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683935    2550 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683939    2550 flags.go:64] FLAG: --help="false"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683942    2550 flags.go:64] FLAG: --hostname-override="ip-10-0-129-227.ec2.internal"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683945    2550 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683948    2550 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683951    2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683954    2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683958    2550 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683961    2550 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683964    2550 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683967    2550 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683970    2550 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683973    2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683977    2550 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683979    2550 flags.go:64] FLAG: --kube-reserved=""
Sep 15 18:50:36.691409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683982    2550 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683985    2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683988    2550 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683991    2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683995    2550 flags.go:64] FLAG: --lock-file=""
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.683998    2550 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684001    2550 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684004    2550 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684010    2550 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684013    2550 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684016    2550 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684018    2550 flags.go:64] FLAG: --logging-format="text"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684021    2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684025    2550 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684028    2550 flags.go:64] FLAG: --manifest-url=""
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684030    2550 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684035    2550 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684038    2550 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684042    2550 flags.go:64] FLAG: --max-pods="110"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684045    2550 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684048    2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684051    2550 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684054    2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684057    2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684060    2550 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 18:50:36.692048 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684064    2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684071    2550 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684074    2550 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684077    2550 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684081    2550 flags.go:64] FLAG: --pod-cidr=""
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684084    2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684089    2550 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684092    2550 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684095    2550 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684098    2550 flags.go:64] FLAG: --port="10250"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684101    2550 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684104    2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02d86bed6f047f7cf"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684108    2550 flags.go:64] FLAG: --qos-reserved=""
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684111    2550 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684114    2550 flags.go:64] FLAG: --register-node="true"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684117    2550 flags.go:64] FLAG: --register-with-taints=""
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684121    2550 flags.go:64] FLAG: --registry-burst="10"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684124    2550 flags.go:64] FLAG: --registry-qps="5"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684127    2550 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684130    2550 flags.go:64] FLAG: --reserved-memory=""
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684134    2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684137    2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684140    2550 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684143    2550 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684146    2550 flags.go:64] FLAG: --runonce="false"
Sep 15 18:50:36.692725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684149    2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684152    2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684155    2550 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684157    2550 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684160    2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684163    2550 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684167    2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684170    2550 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684173    2550 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684176    2550 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684179    2550 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684182    2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684185    2550 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684188    2550 flags.go:64] FLAG: --system-cgroups=""
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684191    2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684197    2550 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684200    2550 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684203    2550 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684207    2550 flags.go:64] FLAG: --tls-min-version=""
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684210    2550 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684213    2550 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684215    2550 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684218    2550 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684221    2550 flags.go:64] FLAG: --v="2"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684225    2550 flags.go:64] FLAG: --version="false"
Sep 15 18:50:36.693356 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684229    2550 flags.go:64] FLAG: --vmodule=""
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684232    2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.684235    2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684375    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684379    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684382    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684385    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684389    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684391    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684394    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684397    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684400    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684402    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684406    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684409    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684412    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684415    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684417    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684420    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684422    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:36.693989 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684426    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684429    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684431    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684448    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684451    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684453    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684456    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684459    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684461    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684464    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684467    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684470    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684472    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684475    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684478    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684482    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684485    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684487    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684490    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684493    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:36.694527 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684495    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684498    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684500    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684503    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684505    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684509    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684512    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684515    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684519    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684522    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684525    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684528    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684531    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684534    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684537    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684539    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684542    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684544    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684548    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:36.695020 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684551    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684554    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684557    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684559    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684561    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684564    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684567    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684570    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684575    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684579    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684581    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684584    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684587    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684591    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684593    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684596    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684599    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684602    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684606    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:36.695511 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684609    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684611    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684614    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684617    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684620    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684622    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684625    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684627    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684630    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684633    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684636    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684638    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684641    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684643    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684646    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684648    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684651    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684654    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684656    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684659    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:36.695975 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684661    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:36.696491 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684665    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:36.696491 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684668    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:36.696491 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.684671    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:36.696491 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.685187    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 18:50:36.696491 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.693638    2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 18:50:36.696491 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.693657    2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 18:50:36.696491 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.694421    2550 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 18:50:36.696491 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.694430    2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 18:50:36.696491 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694515    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:36.696491 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694522    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:36.696491 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694526    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:36.696491 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694530    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:36.696491 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694533    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:36.696491 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694537    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:36.696491 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694539    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694542    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694545    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694548    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694551    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694554    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694556    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694560    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694564    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694567    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694570    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694573    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694576    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694580    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694584    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694587    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694590    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694593    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694596    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:36.696871 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694599    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694602    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694605    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694607    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694611    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694614    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694617    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694620    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694623    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694626    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694628    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694631    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694634    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694637    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694640    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694643    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694645    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694647    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694650    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694654    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:36.697335 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694657    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694660    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694664    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694667    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694670    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694674    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694677    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694680    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694684    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694686    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694689    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694691    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694694    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694696    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694699    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694702    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694705    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694708    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694711    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:36.697900 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694713    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694716    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694719    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694722    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694725    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694727    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694730    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694732    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694735    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694737    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694740    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694742    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694745    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694748    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694750    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694753    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694756    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694759    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694761    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694764    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:36.698376 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694766    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:36.698876 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694769    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:36.698876 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694772    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:36.698876 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694774    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:36.698876 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694777    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:36.698876 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694779    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:36.698876 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694782    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:36.698876 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694784    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:36.698876 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694787    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:36.698876 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694790    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:36.698876 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694793    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:36.698876 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694796    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:36.698876 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694798    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:36.698876 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694801    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:36.698876 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694803    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:36.698876 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.694809    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694939    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694944    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694948    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694951    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694954    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694957    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694959    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694962    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694965    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694967    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694970    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694972    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694975    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694978    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694981    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694985    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694989    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694992    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:36.699257 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694994    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.694998    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695000    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695003    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695006    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695009    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695012    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695015    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695019    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695021    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695024    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695027    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695030    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695033    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695036    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695039    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695043    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695046    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695049    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:36.699779 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695052    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695055    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695058    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695060    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695063    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695066    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695068    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695071    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695073    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695077    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695079    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695082    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695085    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695087    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695090    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695092    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695095    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695097    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695100    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695102    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:36.700251 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695106    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695109    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695112    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695115    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695117    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695120    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695122    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695125    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695128    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695130    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695133    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695135    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695138    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695140    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695143    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695146    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695149    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695152    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695154    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695157    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:36.700819 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695159    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695162    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695165    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695168    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695170    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695173    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695176    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695179    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695182    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695185    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695188    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695190    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695193    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695195    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695198    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695201    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695204    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695207    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695210    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:36.701324 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695212    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:36.702051 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695215    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:36.702051 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:36.695217    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:36.702051 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.695222    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 18:50:36.702051 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.695349    2550 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 18:50:36.702051 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.698244    2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 18:50:36.702051 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.699144    2550 server.go:1019] "Starting client certificate rotation"
Sep 15 18:50:36.702051 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.699240    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 18:50:36.702051 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.699305    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 18:50:36.702051 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.701829    2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 18:50:36.703023 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.703004    2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 18:50:36.712734 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.712716    2550 log.go:25] "Validated CRI v1 runtime API"
Sep 15 18:50:36.715855 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.715827    2550 log.go:25] "Validated CRI v1 image API"
Sep 15 18:50:36.717823 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.717792    2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 18:50:36.722786 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.722759    2550 fs.go:135] Filesystem UUIDs: map[08f04117-59ce-4666-a991-fa004fdb6c8e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fe8a6ca2-24e4-4930-a19e-736908e4dbfd:/dev/nvme0n1p3]
Sep 15 18:50:36.722882 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.722782    2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 18:50:36.728604 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.728478    2550 manager.go:216] Machine: {Timestamp:2026-09-15 18:50:36.727113927 +0000 UTC m=+0.313756521 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100634 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20ba2d883dad014165b0708050d71f SystemUUID:ec20ba2d-883d-ad01-4165-b0708050d71f BootID:f89f715e-394c-4281-be48-800113b3f716 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:eb:bc:62:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:eb:bc:62:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:67:3e:f1:b2:ee Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 18:50:36.728604 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.728591    2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 18:50:36.728790 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.728706    2550 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 18:50:36.728863 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.728844    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 18:50:36.731691 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.731662    2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 18:50:36.731836 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.731695    2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-227.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 18:50:36.732612 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.732602    2550 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 18:50:36.732654 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.732615    2550 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 18:50:36.732654 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.732625    2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 18:50:36.733358 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.733349    2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 18:50:36.734196 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.734186    2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 18:50:36.737771 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.737759    2550 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 18:50:36.737883 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.737874    2550 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 18:50:36.737924 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.737903    2550 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 18:50:36.737924 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.737912    2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 18:50:36.737987 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.737929    2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 18:50:36.737987 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.737938    2550 kubelet.go:398] "Adding apiserver pod source"
Sep 15 18:50:36.737987 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.737947    2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 18:50:36.738298 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.738287    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 18:50:36.738340 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.738305    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 18:50:36.741812 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.741797    2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 18:50:36.744568 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.744553    2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 18:50:36.744624 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.744583    2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 18:50:36.746294 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.746282    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 18:50:36.746330 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.746305    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 18:50:36.746330 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.746321    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 18:50:36.746330 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.746330    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 18:50:36.746422 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.746342    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 18:50:36.746422 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.746355    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 18:50:36.746422 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.746381    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 18:50:36.746422 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.746391    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 18:50:36.746422 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.746398    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 18:50:36.746422 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.746404    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 18:50:36.746422 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.746419    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 18:50:36.746422 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.746425    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 18:50:36.747158 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.747147    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 18:50:36.747197 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.747159    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 18:50:36.749805 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.749769    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-56h7w"
Sep 15 18:50:36.750030 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.750015    2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-227.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 18:50:36.750479 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:36.750459    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 18:50:36.750534 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:36.750462    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 18:50:36.751061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.751049    2550 server.go:1268] "Started kubelet"
Sep 15 18:50:36.751171 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.751138    2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 18:50:36.751216 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.751161    2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 18:50:36.751260 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.751229    2550 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 18:50:36.752123 ip-10-0-129-227 systemd[1]: Started Kubernetes Kubelet.
Sep 15 18:50:36.753241 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.752533    2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 18:50:36.753241 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.753103    2550 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 18:50:36.757925 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.757857    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-56h7w"
Sep 15 18:50:36.759737 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:36.759593    2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 18:50:36.762036 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.762020    2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 18:50:36.762138 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.762086    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 18:50:36.762241 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:36.762226    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-227.ec2.internal\" not found"
Sep 15 18:50:36.762900 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.762881    2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 18:50:36.762900 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.762884    2550 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 18:50:36.763041 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.762915    2550 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 18:50:36.763041 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.763033    2550 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 18:50:36.763127 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.763043    2550 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 18:50:36.763174 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:36.758996    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-227.ec2.internal.18d5932a47bb875f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-227.ec2.internal,UID:ip-10-0-129-227.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-227.ec2.internal,},FirstTimestamp:2026-09-15 18:50:36.751021919 +0000 UTC m=+0.337664514,LastTimestamp:2026-09-15 18:50:36.751021919 +0000 UTC m=+0.337664514,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-227.ec2.internal,}"
Sep 15 18:50:36.763922 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.763897    2550 factory.go:55] Registering systemd factory
Sep 15 18:50:36.763922 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.763917    2550 factory.go:223] Registration of the systemd container factory successfully
Sep 15 18:50:36.764210 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.764191    2550 factory.go:153] Registering CRI-O factory
Sep 15 18:50:36.764210 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.764211    2550 factory.go:223] Registration of the crio container factory successfully
Sep 15 18:50:36.764350 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:36.764226    2550 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 18:50:36.764350 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.764271    2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 18:50:36.764350 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.764289    2550 factory.go:103] Registering Raw factory
Sep 15 18:50:36.764350 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.764306    2550 manager.go:1196] Started watching for new ooms in manager
Sep 15 18:50:36.764735 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.764720    2550 manager.go:349] Starting recovery of all containers
Sep 15 18:50:36.775425 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.775410    2550 manager.go:354] Recovery completed
Sep 15 18:50:36.777149 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.776965    2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:36.781618 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.781606    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:36.783366 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.783347    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:36.783469 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.783387    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:36.783469 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.783402    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:36.784043 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.784028    2550 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 18:50:36.784043 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.784042    2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 18:50:36.784176 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.784059    2550 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 18:50:36.784176 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.784051    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:36.784833 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.784819    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:36.784928 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.784846    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:36.784928 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.784859    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:36.786168 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.786151    2550 policy_none.go:49] "None policy: Start"
Sep 15 18:50:36.786168 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.786166    2550 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 18:50:36.786324 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.786176    2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 18:50:36.788186 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.788170    2550 policy_none.go:47] "Start"
Sep 15 18:50:36.928167 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.827046    2550 manager.go:337] "Starting Device Plugin manager"
Sep 15 18:50:36.928167 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:36.827086    2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 18:50:36.928167 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.827097    2550 server.go:87] "Starting device plugin registration server"
Sep 15 18:50:36.928167 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.827425    2550 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 18:50:36.928167 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.827453    2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 18:50:36.928167 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.827557    2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 18:50:36.928167 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.827645    2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 18:50:36.928167 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.827654    2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 18:50:36.928167 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:36.828356    2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 18:50:36.928167 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:36.828419    2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-227.ec2.internal\" not found"
Sep 15 18:50:36.928167 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.846453    2550 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2584 into the system container \"/system.slice\": <nil>"
Sep 15 18:50:36.928167 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.927714    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:36.930547 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.930527    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:36.930664 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.930559    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:36.930664 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.930578    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:36.930664 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.930604    2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-227.ec2.internal"
Sep 15 18:50:36.938122 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.938102    2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-227.ec2.internal"
Sep 15 18:50:36.938254 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:36.938130    2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-227.ec2.internal\": node \"ip-10-0-129-227.ec2.internal\" not found"
Sep 15 18:50:36.951724 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:36.951698    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-227.ec2.internal\" not found"
Sep 15 18:50:36.977594 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.977565    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 18:50:36.978882 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.978866    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 18:50:36.978969 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.978915    2550 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 18:50:36.978969 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.978938    2550 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 18:50:36.979058 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:36.979037    2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 18:50:36.981279 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:36.981253    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:37.052696 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:37.052662    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-227.ec2.internal\" not found"
Sep 15 18:50:37.080082 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.080055    2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-227.ec2.internal"]
Sep 15 18:50:37.080165 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.080131    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:37.081881 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.081858    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:37.081881 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.081887    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:37.082029 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.081896    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:37.084244 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.084230    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:37.084350 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.084327    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal"
Sep 15 18:50:37.084399 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.084365    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:37.085286 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.085269    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:37.085374 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.085296    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:37.085374 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.085306    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:37.085374 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.085273    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:37.085374 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.085365    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:37.085374 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.085375    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:37.087684 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.087669    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-227.ec2.internal"
Sep 15 18:50:37.087740 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.087695    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:37.088679 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.088665    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:37.088744 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.088695    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:37.088744 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.088707    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:37.116256 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:37.116235    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-227.ec2.internal\" not found" node="ip-10-0-129-227.ec2.internal"
Sep 15 18:50:37.120883 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:37.120868    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-227.ec2.internal\" not found" node="ip-10-0-129-227.ec2.internal"
Sep 15 18:50:37.153289 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:37.153257    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-227.ec2.internal\" not found"
Sep 15 18:50:37.253550 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:37.253471    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-227.ec2.internal\" not found"
Sep 15 18:50:37.264872 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.264844    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3d6ec7431a5c04a615ddb019fc77a6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal\" (UID: \"b3d6ec7431a5c04a615ddb019fc77a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal"
Sep 15 18:50:37.264961 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.264895    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/527678610c0f9e9b262ca08d3e160568-config\") pod \"kube-apiserver-proxy-ip-10-0-129-227.ec2.internal\" (UID: \"527678610c0f9e9b262ca08d3e160568\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-227.ec2.internal"
Sep 15 18:50:37.264961 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.264925    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b3d6ec7431a5c04a615ddb019fc77a6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal\" (UID: \"b3d6ec7431a5c04a615ddb019fc77a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal"
Sep 15 18:50:37.354023 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:37.353975    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-227.ec2.internal\" not found"
Sep 15 18:50:37.365357 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.365333    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b3d6ec7431a5c04a615ddb019fc77a6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal\" (UID: \"b3d6ec7431a5c04a615ddb019fc77a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal"
Sep 15 18:50:37.365357 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.365344    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b3d6ec7431a5c04a615ddb019fc77a6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal\" (UID: \"b3d6ec7431a5c04a615ddb019fc77a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal"
Sep 15 18:50:37.365509 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.365376    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3d6ec7431a5c04a615ddb019fc77a6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal\" (UID: \"b3d6ec7431a5c04a615ddb019fc77a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal"
Sep 15 18:50:37.365509 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.365394    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/527678610c0f9e9b262ca08d3e160568-config\") pod \"kube-apiserver-proxy-ip-10-0-129-227.ec2.internal\" (UID: \"527678610c0f9e9b262ca08d3e160568\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-227.ec2.internal"
Sep 15 18:50:37.365509 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.365419    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/527678610c0f9e9b262ca08d3e160568-config\") pod \"kube-apiserver-proxy-ip-10-0-129-227.ec2.internal\" (UID: \"527678610c0f9e9b262ca08d3e160568\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-227.ec2.internal"
Sep 15 18:50:37.365509 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:37.365417    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3d6ec7431a5c04a615ddb019fc77a6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal\" (UID: \"b3d6ec7431a5c04a615ddb019fc77a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal"
Sep 15 18:50:38.729388 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:38.729343    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal"
Sep 15 18:50:38.734669 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:38.734511    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-227.ec2.internal"
Sep 15 18:50:38.763996 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:38.763965    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-227.ec2.internal\" not found"
Sep 15 18:50:38.864575 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:38.864481    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-227.ec2.internal\" not found"
Sep 15 18:50:38.954159 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:38.954137    2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:38.973132 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:38.973108    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-227.ec2.internal"
Sep 15 18:50:38.983189 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:38.983170    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 18:50:38.985099 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:38.985085    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal"
Sep 15 18:50:38.997855 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:38.997832    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 18:50:39.009005 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.008985    2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 18:50:39.009096 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.009072    2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 18:50:39.009141 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.009087    2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 18:50:39.009141 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.009088    2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 18:50:39.048326 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.048287    2550 apiserver.go:52] "Watching apiserver"
Sep 15 18:50:39.062370 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.062341    2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 18:50:39.063861 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.063837    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-rnhds","kube-system/konnectivity-agent-fnczv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal","openshift-multus/network-metrics-daemon-9tr5r","openshift-network-diagnostics/network-check-target-8jngj","openshift-network-operator/iptables-alerter-b8g7s","openshift-ovn-kubernetes/ovnkube-node-qmzls","kube-system/global-pull-secret-syncer-vgmvv","kube-system/kube-apiserver-proxy-ip-10-0-129-227.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs","openshift-cluster-node-tuning-operator/tuned-gb7rw","openshift-dns/node-resolver-5n5wz","openshift-image-registry/node-ca-776nl","openshift-multus/multus-additional-cni-plugins-wvsmg"]
Sep 15 18:50:39.066332 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.066295    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.068383 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.068362    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fnczv"
Sep 15 18:50:39.068499 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.068430    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 18:50:39.068666 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.068648    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 18:50:39.068748 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.068652    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 18:50:39.068850 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.068836    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 18:50:39.068909 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.068896    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h8txg\""
Sep 15 18:50:39.070561 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.070544    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:39.070633 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.070608    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:50:39.070986 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.070970    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 18:50:39.071049 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.071032    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 18:50:39.071105 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.071091    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jzfs5\""
Sep 15 18:50:39.073597 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.072595    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 18:50:39.074008 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.073981    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:39.074102 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.074064    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:50:39.074620 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.074570    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 18:45:38 +0000 UTC" deadline="2028-06-30 12:02:35.768014146 +0000 UTC"
Sep 15 18:50:39.074680 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.074619    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15689h11m56.693397318s"
Sep 15 18:50:39.076519 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.076502    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b8g7s"
Sep 15 18:50:39.077700 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.077678    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjsrq\" (UniqueName: \"kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq\") pod \"network-check-target-8jngj\" (UID: \"540b4731-6750-43d4-96b3-759aec9a4eee\") " pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:39.077794 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.077715    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-run-netns\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.077794 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.077741    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-var-lib-cni-bin\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.077794 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.077764    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-multus-conf-dir\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.077794 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.077789    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-multus-daemon-config\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.077994 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.077811    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-run-multus-certs\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.077994 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.077860    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-system-cni-dir\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.077994 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.077914    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-multus-cni-dir\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.077994 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.077946    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-cnibin\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.077994 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.077971    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-multus-socket-dir-parent\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.078230 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.077996    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-etc-kubernetes\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.078230 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.078018    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqrcc\" (UniqueName: \"kubernetes.io/projected/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-kube-api-access-sqrcc\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.078230 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.078041    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-var-lib-cni-multus\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.078230 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.078064    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-var-lib-kubelet\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.078230 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.078086    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-hostroot\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.078230 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.078108    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3707c81-43dd-4142-9c7b-82b12577c423-agent-certs\") pod \"konnectivity-agent-fnczv\" (UID: \"b3707c81-43dd-4142-9c7b-82b12577c423\") " pod="kube-system/konnectivity-agent-fnczv"
Sep 15 18:50:39.078230 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.078133    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs\") pod \"network-metrics-daemon-9tr5r\" (UID: \"66062b2c-ba98-4b6a-bc2c-656a6c7c339b\") " pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:39.078230 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.078158    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk465\" (UniqueName: \"kubernetes.io/projected/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-kube-api-access-zk465\") pod \"network-metrics-daemon-9tr5r\" (UID: \"66062b2c-ba98-4b6a-bc2c-656a6c7c339b\") " pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:39.078230 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.078181    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-cni-binary-copy\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.078230 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.078203    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-run-k8s-cni-cncf-io\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.078230 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.078225    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3707c81-43dd-4142-9c7b-82b12577c423-konnectivity-ca\") pod \"konnectivity-agent-fnczv\" (UID: \"b3707c81-43dd-4142-9c7b-82b12577c423\") " pod="kube-system/konnectivity-agent-fnczv"
Sep 15 18:50:39.078884 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.078870    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.079236 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.079203    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j4xtd\""
Sep 15 18:50:39.079236 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.079221    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 18:50:39.079236 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.079228    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 18:50:39.079540 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.079526    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:50:39.081005 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.080984    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 18:50:39.081087 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.081005    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 18:50:39.081087 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.081078    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n2zfj\""
Sep 15 18:50:39.081280 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.081265    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 18:50:39.081376 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.081324    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 18:50:39.081432 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.081391    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 18:50:39.081481 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.081427    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:39.081481 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.081450    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 18:50:39.081563 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.081496    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:50:39.083808 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.083792    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.084441 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.084423    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 18:50:39.085635 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.085621    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 18:50:39.085722 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.085642    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p248f\""
Sep 15 18:50:39.085773 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.085725    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 18:50:39.085817 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.085777    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 18:50:39.085983 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.085970    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.087770 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.087755    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2nvrr\""
Sep 15 18:50:39.087842 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.087817    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:50:39.087885 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.087868    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 18:50:39.088167 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.088155    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5n5wz"
Sep 15 18:50:39.090132 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.090118    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 18:50:39.090183 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.090152    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 18:50:39.090223 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.090152    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-85sts\""
Sep 15 18:50:39.090371 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.090352    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-776nl"
Sep 15 18:50:39.092229 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.092212    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 18:50:39.092332 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.092262    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 18:50:39.092407 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.092394    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 18:50:39.092539 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.092525    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rdntb\""
Sep 15 18:50:39.092645 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.092630    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.094536 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.094520    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 18:50:39.094623 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.094575    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 18:50:39.094707 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.094690    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qp82p\""
Sep 15 18:50:39.107999 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.107974    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rqwvq"
Sep 15 18:50:39.113840 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.113824    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rqwvq"
Sep 15 18:50:39.173690 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.173664    2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 18:50:39.178692 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.178668    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-cnibin\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.178810 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.178699    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.178810 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.178715    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-run-ovn-kubernetes\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.178810 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.178737    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-sysconfig\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.178810 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.178760    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45a116a2-6657-4785-8769-056a5708db32-serviceca\") pod \"node-ca-776nl\" (UID: \"45a116a2-6657-4785-8769-056a5708db32\") " pod="openshift-image-registry/node-ca-776nl"
Sep 15 18:50:39.178810 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.178782    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-dbus\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:39.178810 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.178797    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-system-cni-dir\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.178810 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.178811    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-slash\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.179115 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.178825    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-tuned\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.179115 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.178860    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-hostroot\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.179115 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.178911    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-hostroot\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.179115 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.178944    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs\") pod \"network-metrics-daemon-9tr5r\" (UID: \"66062b2c-ba98-4b6a-bc2c-656a6c7c339b\") " pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:39.179115 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.178993    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-kubelet\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.179115 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179019    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-var-lib-openvswitch\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.179115 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179046    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.179115 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.179053    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:39.179115 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179071    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-ovnkube-config\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.179115 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179095    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-env-overrides\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.179115 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.179110    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs podName:66062b2c-ba98-4b6a-bc2c-656a6c7c339b nodeName:}" failed. No retries permitted until 2026-09-15 18:50:39.679094372 +0000 UTC m=+1.955863207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs") pod "network-metrics-daemon-9tr5r" (UID: "66062b2c-ba98-4b6a-bc2c-656a6c7c339b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179131    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-sys\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179156    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-run-openvswitch\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179169    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-run\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179182    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45a116a2-6657-4785-8769-056a5708db32-host\") pod \"node-ca-776nl\" (UID: \"45a116a2-6657-4785-8769-056a5708db32\") " pod="openshift-image-registry/node-ca-776nl"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179199    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-run-netns\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179214    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-var-lib-cni-bin\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179228    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-multus-conf-dir\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179244    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-multus-daemon-config\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179266    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-run-systemd\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179299    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-multus-conf-dir\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179298    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-var-lib-cni-bin\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179305    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-run-netns\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179335    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-sys-fs\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179357    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fd5a36a-c351-4e64-8b75-abf7e06279a0-tmp-dir\") pod \"node-resolver-5n5wz\" (UID: \"9fd5a36a-c351-4e64-8b75-abf7e06279a0\") " pod="openshift-dns/node-resolver-5n5wz"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179374    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-multus-cni-dir\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179389    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk465\" (UniqueName: \"kubernetes.io/projected/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-kube-api-access-zk465\") pod \"network-metrics-daemon-9tr5r\" (UID: \"66062b2c-ba98-4b6a-bc2c-656a6c7c339b\") " pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:39.179603 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179412    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxgcl\" (UniqueName: \"kubernetes.io/projected/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-kube-api-access-rxgcl\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179429    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2drg\" (UniqueName: \"kubernetes.io/projected/722d9d5c-0836-40ea-a990-f60b9eff5b5c-kube-api-access-l2drg\") pod \"iptables-alerter-b8g7s\" (UID: \"722d9d5c-0836-40ea-a990-f60b9eff5b5c\") " pod="openshift-network-operator/iptables-alerter-b8g7s"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179444    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-ovnkube-script-lib\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179456    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-multus-cni-dir\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179463    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-etc-selinux\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179504    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-host\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179544    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-var-lib-kubelet\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179579    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-var-lib-kubelet\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179607    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179632    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/722d9d5c-0836-40ea-a990-f60b9eff5b5c-host-slash\") pod \"iptables-alerter-b8g7s\" (UID: \"722d9d5c-0836-40ea-a990-f60b9eff5b5c\") " pod="openshift-network-operator/iptables-alerter-b8g7s"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179656    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-run-netns\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179681    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179709    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-cni-binary-copy\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179734    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-run-k8s-cni-cncf-io\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179758    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/722d9d5c-0836-40ea-a990-f60b9eff5b5c-iptables-alerter-script\") pod \"iptables-alerter-b8g7s\" (UID: \"722d9d5c-0836-40ea-a990-f60b9eff5b5c\") " pod="openshift-network-operator/iptables-alerter-b8g7s"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179783    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4j6q\" (UniqueName: \"kubernetes.io/projected/f4393efb-59bf-4bd4-9cf4-3ab42788041a-kube-api-access-j4j6q\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179806    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-systemd\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.180246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179813    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-run-k8s-cni-cncf-io\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179831    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-kubernetes\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179858    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-system-cni-dir\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179892    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqrcc\" (UniqueName: \"kubernetes.io/projected/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-kube-api-access-sqrcc\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179915    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179939    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.179974    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-system-cni-dir\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180040    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-lib-modules\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180075    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-tmp\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180109    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3707c81-43dd-4142-9c7b-82b12577c423-agent-certs\") pod \"konnectivity-agent-fnczv\" (UID: \"b3707c81-43dd-4142-9c7b-82b12577c423\") " pod="kube-system/konnectivity-agent-fnczv"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180128    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-node-log\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180142    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-cni-bin\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180160    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-cni-netd\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180185    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3707c81-43dd-4142-9c7b-82b12577c423-konnectivity-ca\") pod \"konnectivity-agent-fnczv\" (UID: \"b3707c81-43dd-4142-9c7b-82b12577c423\") " pod="kube-system/konnectivity-agent-fnczv"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180205    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjsrq\" (UniqueName: \"kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq\") pod \"network-check-target-8jngj\" (UID: \"540b4731-6750-43d4-96b3-759aec9a4eee\") " pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180219    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-systemd-units\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180233    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-etc-openvswitch\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.180990 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180344    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-sysctl-d\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180370    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-kubelet-config\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180397    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mxnl\" (UniqueName: \"kubernetes.io/projected/9fd5a36a-c351-4e64-8b75-abf7e06279a0-kube-api-access-9mxnl\") pod \"node-resolver-5n5wz\" (UID: \"9fd5a36a-c351-4e64-8b75-abf7e06279a0\") " pod="openshift-dns/node-resolver-5n5wz"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180422    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-ovn-node-metrics-cert\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180444    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-run-multus-certs\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180461    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-run-ovn\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180481    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-log-socket\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180504    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-modprobe-d\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180513    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-run-multus-certs\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180527    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-sysctl-conf\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180551    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-cnibin\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180567    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-multus-daemon-config\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180570    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-multus-socket-dir-parent\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180603    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-cni-binary-copy\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180618    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-cnibin\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180626    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-multus-socket-dir-parent\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180630    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-etc-kubernetes\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.181480 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180662    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-registration-dir\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.181989 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180687    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-867gg\" (UniqueName: \"kubernetes.io/projected/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-kube-api-access-867gg\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.181989 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180707    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-etc-kubernetes\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.181989 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180710    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x298\" (UniqueName: \"kubernetes.io/projected/45a116a2-6657-4785-8769-056a5708db32-kube-api-access-5x298\") pod \"node-ca-776nl\" (UID: \"45a116a2-6657-4785-8769-056a5708db32\") " pod="openshift-image-registry/node-ca-776nl"
Sep 15 18:50:39.181989 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180738    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-var-lib-cni-multus\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.181989 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180762    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-var-lib-kubelet\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.181989 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180780    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-host-var-lib-cni-multus\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.181989 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180786    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fd5a36a-c351-4e64-8b75-abf7e06279a0-hosts-file\") pod \"node-resolver-5n5wz\" (UID: \"9fd5a36a-c351-4e64-8b75-abf7e06279a0\") " pod="openshift-dns/node-resolver-5n5wz"
Sep 15 18:50:39.181989 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180811    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lvxk\" (UniqueName: \"kubernetes.io/projected/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-kube-api-access-4lvxk\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.181989 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180835    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-socket-dir\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.181989 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180857    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-device-dir\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.181989 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.180895    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:39.182304 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.182213    2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 18:50:39.182378 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.182364    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3707c81-43dd-4142-9c7b-82b12577c423-konnectivity-ca\") pod \"konnectivity-agent-fnczv\" (UID: \"b3707c81-43dd-4142-9c7b-82b12577c423\") " pod="kube-system/konnectivity-agent-fnczv"
Sep 15 18:50:39.185477 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.185448    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3707c81-43dd-4142-9c7b-82b12577c423-agent-certs\") pod \"konnectivity-agent-fnczv\" (UID: \"b3707c81-43dd-4142-9c7b-82b12577c423\") " pod="kube-system/konnectivity-agent-fnczv"
Sep 15 18:50:39.185821 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.185805    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:39.185868 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.185827    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:39.185868 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.185837    2550 projected.go:196] Error preparing data for projected volume kube-api-access-kjsrq for pod openshift-network-diagnostics/network-check-target-8jngj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:39.185929 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.185918    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq podName:540b4731-6750-43d4-96b3-759aec9a4eee nodeName:}" failed. No retries permitted until 2026-09-15 18:50:39.685895073 +0000 UTC m=+1.962663937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kjsrq" (UniqueName: "kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq") pod "network-check-target-8jngj" (UID: "540b4731-6750-43d4-96b3-759aec9a4eee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:39.187781 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.187754    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqrcc\" (UniqueName: \"kubernetes.io/projected/f07817f7-e8cf-4ba0-a211-9e3d85ee4f79-kube-api-access-sqrcc\") pod \"multus-rnhds\" (UID: \"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79\") " pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.187868 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.187774    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk465\" (UniqueName: \"kubernetes.io/projected/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-kube-api-access-zk465\") pod \"network-metrics-daemon-9tr5r\" (UID: \"66062b2c-ba98-4b6a-bc2c-656a6c7c339b\") " pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:39.251279 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.251253    2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:39.281659 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281628    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-systemd-units\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.281851 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281672    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-etc-openvswitch\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.281851 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281697    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-sysctl-d\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.281851 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281722    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-kubelet-config\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:39.281851 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281746    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mxnl\" (UniqueName: \"kubernetes.io/projected/9fd5a36a-c351-4e64-8b75-abf7e06279a0-kube-api-access-9mxnl\") pod \"node-resolver-5n5wz\" (UID: \"9fd5a36a-c351-4e64-8b75-abf7e06279a0\") " pod="openshift-dns/node-resolver-5n5wz"
Sep 15 18:50:39.281851 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281747    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-systemd-units\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.281851 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281747    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-etc-openvswitch\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.281851 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281824    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-ovn-node-metrics-cert\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.281851 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281841    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-kubelet-config\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:39.281851 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281849    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-run-ovn\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.282274 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281870    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-sysctl-d\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.282274 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281886    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-run-ovn\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.282274 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281890    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-log-socket\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.282274 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281913    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-log-socket\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.282274 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281924    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-modprobe-d\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.282274 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281951    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-sysctl-conf\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.282274 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.281995    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-registration-dir\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.282274 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282055    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-registration-dir\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.282274 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282091    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-867gg\" (UniqueName: \"kubernetes.io/projected/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-kube-api-access-867gg\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.282274 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282109    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-modprobe-d\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.282274 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282127    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x298\" (UniqueName: \"kubernetes.io/projected/45a116a2-6657-4785-8769-056a5708db32-kube-api-access-5x298\") pod \"node-ca-776nl\" (UID: \"45a116a2-6657-4785-8769-056a5708db32\") " pod="openshift-image-registry/node-ca-776nl"
Sep 15 18:50:39.282274 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282113    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-sysctl-conf\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.282715 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282298    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-var-lib-kubelet\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.282715 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282662    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-var-lib-kubelet\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.282791 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282713    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fd5a36a-c351-4e64-8b75-abf7e06279a0-hosts-file\") pod \"node-resolver-5n5wz\" (UID: \"9fd5a36a-c351-4e64-8b75-abf7e06279a0\") " pod="openshift-dns/node-resolver-5n5wz"
Sep 15 18:50:39.282791 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282740    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lvxk\" (UniqueName: \"kubernetes.io/projected/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-kube-api-access-4lvxk\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.282791 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282764    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-socket-dir\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.282939 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282823    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fd5a36a-c351-4e64-8b75-abf7e06279a0-hosts-file\") pod \"node-resolver-5n5wz\" (UID: \"9fd5a36a-c351-4e64-8b75-abf7e06279a0\") " pod="openshift-dns/node-resolver-5n5wz"
Sep 15 18:50:39.282939 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282862    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-device-dir\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.282939 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282891    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:39.282939 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282920    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-cnibin\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.282939 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282926    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-socket-dir\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.283172 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282946    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.283172 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282971    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-device-dir\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.283172 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282988    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-cnibin\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.283172 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.282976    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-run-ovn-kubernetes\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.283172 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283024    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-sysconfig\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.283172 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283029    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-run-ovn-kubernetes\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.283172 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.283045    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:39.283172 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283073    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-sysconfig\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.283172 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283053    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45a116a2-6657-4785-8769-056a5708db32-serviceca\") pod \"node-ca-776nl\" (UID: \"45a116a2-6657-4785-8769-056a5708db32\") " pod="openshift-image-registry/node-ca-776nl"
Sep 15 18:50:39.283172 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.283112    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret podName:f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae nodeName:}" failed. No retries permitted until 2026-09-15 18:50:39.783091682 +0000 UTC m=+2.059860516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret") pod "global-pull-secret-syncer-vgmvv" (UID: "f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:39.283172 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283147    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-dbus\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283177    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-system-cni-dir\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283206    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-slash\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283233    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-tuned\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283303    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-kubelet\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283353    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-var-lib-openvswitch\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283389    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283403    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-dbus\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283421    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-ovnkube-config\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283444    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-env-overrides\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283468    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-sys\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283474    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-slash\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283493    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-run-openvswitch\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283500    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283516    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-run\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283538    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45a116a2-6657-4785-8769-056a5708db32-host\") pod \"node-ca-776nl\" (UID: \"45a116a2-6657-4785-8769-056a5708db32\") " pod="openshift-image-registry/node-ca-776nl"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283544    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-var-lib-openvswitch\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.283695 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283354    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-system-cni-dir\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283467    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45a116a2-6657-4785-8769-056a5708db32-serviceca\") pod \"node-ca-776nl\" (UID: \"45a116a2-6657-4785-8769-056a5708db32\") " pod="openshift-image-registry/node-ca-776nl"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283568    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-run-systemd\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283580    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283446    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-kubelet\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283661    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45a116a2-6657-4785-8769-056a5708db32-host\") pod \"node-ca-776nl\" (UID: \"45a116a2-6657-4785-8769-056a5708db32\") " pod="openshift-image-registry/node-ca-776nl"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283592    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-sys-fs\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283877    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-run-openvswitch\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283881    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-run\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.283963    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-sys\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284016    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fd5a36a-c351-4e64-8b75-abf7e06279a0-tmp-dir\") pod \"node-resolver-5n5wz\" (UID: \"9fd5a36a-c351-4e64-8b75-abf7e06279a0\") " pod="openshift-dns/node-resolver-5n5wz"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284071    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-run-systemd\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284109    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxgcl\" (UniqueName: \"kubernetes.io/projected/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-kube-api-access-rxgcl\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284150    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-sys-fs\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284148    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2drg\" (UniqueName: \"kubernetes.io/projected/722d9d5c-0836-40ea-a990-f60b9eff5b5c-kube-api-access-l2drg\") pod \"iptables-alerter-b8g7s\" (UID: \"722d9d5c-0836-40ea-a990-f60b9eff5b5c\") " pod="openshift-network-operator/iptables-alerter-b8g7s"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284213    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-ovnkube-script-lib\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284249    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-etc-selinux\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.284444 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284281    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-host\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284335    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284354    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fd5a36a-c351-4e64-8b75-abf7e06279a0-tmp-dir\") pod \"node-resolver-5n5wz\" (UID: \"9fd5a36a-c351-4e64-8b75-abf7e06279a0\") " pod="openshift-dns/node-resolver-5n5wz"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284371    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/722d9d5c-0836-40ea-a990-f60b9eff5b5c-host-slash\") pod \"iptables-alerter-b8g7s\" (UID: \"722d9d5c-0836-40ea-a990-f60b9eff5b5c\") " pod="openshift-network-operator/iptables-alerter-b8g7s"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284410    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-run-netns\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284445    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284520    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284941    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/722d9d5c-0836-40ea-a990-f60b9eff5b5c-host-slash\") pod \"iptables-alerter-b8g7s\" (UID: \"722d9d5c-0836-40ea-a990-f60b9eff5b5c\") " pod="openshift-network-operator/iptables-alerter-b8g7s"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.284993    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/722d9d5c-0836-40ea-a990-f60b9eff5b5c-iptables-alerter-script\") pod \"iptables-alerter-b8g7s\" (UID: \"722d9d5c-0836-40ea-a990-f60b9eff5b5c\") " pod="openshift-network-operator/iptables-alerter-b8g7s"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285034    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4j6q\" (UniqueName: \"kubernetes.io/projected/f4393efb-59bf-4bd4-9cf4-3ab42788041a-kube-api-access-j4j6q\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285050    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-ovnkube-script-lib\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285071    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-systemd\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285108    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-kubernetes\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285125    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-run-netns\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285107    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285166    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4393efb-59bf-4bd4-9cf4-3ab42788041a-etc-selinux\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285167    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.285245 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285214    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285243    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-host\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285249    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-lib-modules\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285718    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285718    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-kubernetes\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285772    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/722d9d5c-0836-40ea-a990-f60b9eff5b5c-iptables-alerter-script\") pod \"iptables-alerter-b8g7s\" (UID: \"722d9d5c-0836-40ea-a990-f60b9eff5b5c\") " pod="openshift-network-operator/iptables-alerter-b8g7s"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285779    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-tmp\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285783    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-ovn-node-metrics-cert\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285840    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-node-log\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285868    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285914    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-env-overrides\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285918    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-node-log\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285929    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-cni-bin\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285873    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-cni-bin\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285980    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-systemd\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.285981    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-cni-netd\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.286061 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.286022    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-host-cni-netd\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.286853 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.286110    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-lib-modules\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.286853 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.286556    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-ovnkube-config\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.290980 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.290521    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-etc-tuned\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.291818 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.291795    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-tmp\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.292171 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.292147    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-867gg\" (UniqueName: \"kubernetes.io/projected/97ed5cdc-1861-49ab-b028-1dc539ea4ef1-kube-api-access-867gg\") pod \"tuned-gb7rw\" (UID: \"97ed5cdc-1861-49ab-b028-1dc539ea4ef1\") " pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.292534 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.292503    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lvxk\" (UniqueName: \"kubernetes.io/projected/d019fddd-e527-4f2f-80f7-37ecf6e5ca20-kube-api-access-4lvxk\") pod \"ovnkube-node-qmzls\" (UID: \"d019fddd-e527-4f2f-80f7-37ecf6e5ca20\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.292627 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.292574    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x298\" (UniqueName: \"kubernetes.io/projected/45a116a2-6657-4785-8769-056a5708db32-kube-api-access-5x298\") pod \"node-ca-776nl\" (UID: \"45a116a2-6657-4785-8769-056a5708db32\") " pod="openshift-image-registry/node-ca-776nl"
Sep 15 18:50:39.292892 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.292866    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mxnl\" (UniqueName: \"kubernetes.io/projected/9fd5a36a-c351-4e64-8b75-abf7e06279a0-kube-api-access-9mxnl\") pod \"node-resolver-5n5wz\" (UID: \"9fd5a36a-c351-4e64-8b75-abf7e06279a0\") " pod="openshift-dns/node-resolver-5n5wz"
Sep 15 18:50:39.293010 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.292949    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxgcl\" (UniqueName: \"kubernetes.io/projected/0dfbf315-3e43-45e4-9a05-4b64c8dbb36c-kube-api-access-rxgcl\") pod \"multus-additional-cni-plugins-wvsmg\" (UID: \"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c\") " pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.293516 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.293494    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4j6q\" (UniqueName: \"kubernetes.io/projected/f4393efb-59bf-4bd4-9cf4-3ab42788041a-kube-api-access-j4j6q\") pod \"aws-ebs-csi-driver-node-r9bgs\" (UID: \"f4393efb-59bf-4bd4-9cf4-3ab42788041a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.294403 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.294380    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2drg\" (UniqueName: \"kubernetes.io/projected/722d9d5c-0836-40ea-a990-f60b9eff5b5c-kube-api-access-l2drg\") pod \"iptables-alerter-b8g7s\" (UID: \"722d9d5c-0836-40ea-a990-f60b9eff5b5c\") " pod="openshift-network-operator/iptables-alerter-b8g7s"
Sep 15 18:50:39.372761 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:39.372691    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527678610c0f9e9b262ca08d3e160568.slice/crio-df34951f77fa8110e82443927ecefc249d58f53b5301fb11ee262811578bd9be WatchSource:0}: Error finding container df34951f77fa8110e82443927ecefc249d58f53b5301fb11ee262811578bd9be: Status 404 returned error can't find the container with id df34951f77fa8110e82443927ecefc249d58f53b5301fb11ee262811578bd9be
Sep 15 18:50:39.378896 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.378881    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 18:50:39.403732 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.403702    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rnhds"
Sep 15 18:50:39.414056 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:39.414030    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07817f7_e8cf_4ba0_a211_9e3d85ee4f79.slice/crio-58359e2e244b900c7112d9b15895967625e14cc5f0f3a32cc1dc6fc0661da839 WatchSource:0}: Error finding container 58359e2e244b900c7112d9b15895967625e14cc5f0f3a32cc1dc6fc0661da839: Status 404 returned error can't find the container with id 58359e2e244b900c7112d9b15895967625e14cc5f0f3a32cc1dc6fc0661da839
Sep 15 18:50:39.417111 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.417093    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fnczv"
Sep 15 18:50:39.425686 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:39.425665    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3707c81_43dd_4142_9c7b_82b12577c423.slice/crio-c8eb5348702c530e8c2e6fafaea5f80534807070e7b7a41705c79ba512d895b0 WatchSource:0}: Error finding container c8eb5348702c530e8c2e6fafaea5f80534807070e7b7a41705c79ba512d895b0: Status 404 returned error can't find the container with id c8eb5348702c530e8c2e6fafaea5f80534807070e7b7a41705c79ba512d895b0
Sep 15 18:50:39.440151 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.440134    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b8g7s"
Sep 15 18:50:39.448105 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:39.448082    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722d9d5c_0836_40ea_a990_f60b9eff5b5c.slice/crio-c6dd3f4d0e090d9a9ba5be0b32052075d78cd5f3690c0bdd64a13e621774099a WatchSource:0}: Error finding container c6dd3f4d0e090d9a9ba5be0b32052075d78cd5f3690c0bdd64a13e621774099a: Status 404 returned error can't find the container with id c6dd3f4d0e090d9a9ba5be0b32052075d78cd5f3690c0bdd64a13e621774099a
Sep 15 18:50:39.457004 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.456985    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:50:39.464898 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:39.464876    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd019fddd_e527_4f2f_80f7_37ecf6e5ca20.slice/crio-21082806340f2010efaa27f763774c39efaa21ddd82f4242fec038b807827ef3 WatchSource:0}: Error finding container 21082806340f2010efaa27f763774c39efaa21ddd82f4242fec038b807827ef3: Status 404 returned error can't find the container with id 21082806340f2010efaa27f763774c39efaa21ddd82f4242fec038b807827ef3
Sep 15 18:50:39.473787 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.473768    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs"
Sep 15 18:50:39.481742 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:39.481718    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4393efb_59bf_4bd4_9cf4_3ab42788041a.slice/crio-93c6edbfc1e7fc8a23c2333a375a601111e13f83dbcc9e5fad60b02780f6aa3c WatchSource:0}: Error finding container 93c6edbfc1e7fc8a23c2333a375a601111e13f83dbcc9e5fad60b02780f6aa3c: Status 404 returned error can't find the container with id 93c6edbfc1e7fc8a23c2333a375a601111e13f83dbcc9e5fad60b02780f6aa3c
Sep 15 18:50:39.494592 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.494571    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gb7rw"
Sep 15 18:50:39.502419 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:39.502396    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ed5cdc_1861_49ab_b028_1dc539ea4ef1.slice/crio-1d8f372292a1657bc090aeb0d37edef02a9db85941a010d621b9bec06c7ca8c0 WatchSource:0}: Error finding container 1d8f372292a1657bc090aeb0d37edef02a9db85941a010d621b9bec06c7ca8c0: Status 404 returned error can't find the container with id 1d8f372292a1657bc090aeb0d37edef02a9db85941a010d621b9bec06c7ca8c0
Sep 15 18:50:39.508128 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.508107    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5n5wz"
Sep 15 18:50:39.525726 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.525607    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-776nl"
Sep 15 18:50:39.528040 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:39.528016    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd5a36a_c351_4e64_8b75_abf7e06279a0.slice/crio-bdbfcc2864c81650a7f6ba4800a7a8ef039d0404b337dff55f222d72d1eabd7b WatchSource:0}: Error finding container bdbfcc2864c81650a7f6ba4800a7a8ef039d0404b337dff55f222d72d1eabd7b: Status 404 returned error can't find the container with id bdbfcc2864c81650a7f6ba4800a7a8ef039d0404b337dff55f222d72d1eabd7b
Sep 15 18:50:39.528836 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:39.528810    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d6ec7431a5c04a615ddb019fc77a6f.slice/crio-0b12dafc4bcb44bfa9dca202b202f0d5fb1961120ea0a11f64bc012cfa114007 WatchSource:0}: Error finding container 0b12dafc4bcb44bfa9dca202b202f0d5fb1961120ea0a11f64bc012cfa114007: Status 404 returned error can't find the container with id 0b12dafc4bcb44bfa9dca202b202f0d5fb1961120ea0a11f64bc012cfa114007
Sep 15 18:50:39.529671 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.529648    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wvsmg"
Sep 15 18:50:39.538004 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:39.537985    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a116a2_6657_4785_8769_056a5708db32.slice/crio-75e9271e82e516287a7122bde4b7dad390ff0f8c015fb88b296a9cd5ef1b8c26 WatchSource:0}: Error finding container 75e9271e82e516287a7122bde4b7dad390ff0f8c015fb88b296a9cd5ef1b8c26: Status 404 returned error can't find the container with id 75e9271e82e516287a7122bde4b7dad390ff0f8c015fb88b296a9cd5ef1b8c26
Sep 15 18:50:39.539929 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:50:39.539897    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dfbf315_3e43_45e4_9a05_4b64c8dbb36c.slice/crio-11abfaaa83a99412de1df0b0fb4f7a8e186ae2ba7cf8aaac1931c626c03b575e WatchSource:0}: Error finding container 11abfaaa83a99412de1df0b0fb4f7a8e186ae2ba7cf8aaac1931c626c03b575e: Status 404 returned error can't find the container with id 11abfaaa83a99412de1df0b0fb4f7a8e186ae2ba7cf8aaac1931c626c03b575e
Sep 15 18:50:39.541588 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.541566    2550 kuberuntime_manager.go:1459] "Unhandled Error" err="init container egress-router-binary-copy start failed in pod multus-additional-cni-plugins-wvsmg_openshift-multus(0dfbf315-3e43-45e4-9a05-4b64c8dbb36c): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 15 18:50:39.541657 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.541599    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-wvsmg" podUID="0dfbf315-3e43-45e4-9a05-4b64c8dbb36c"
Sep 15 18:50:39.688172 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.688082    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs\") pod \"network-metrics-daemon-9tr5r\" (UID: \"66062b2c-ba98-4b6a-bc2c-656a6c7c339b\") " pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:39.688172 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.688131    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjsrq\" (UniqueName: \"kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq\") pod \"network-check-target-8jngj\" (UID: \"540b4731-6750-43d4-96b3-759aec9a4eee\") " pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:39.688376 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.688227    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:39.688376 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.688237    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:39.688376 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.688249    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:39.688376 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.688258    2550 projected.go:196] Error preparing data for projected volume kube-api-access-kjsrq for pod openshift-network-diagnostics/network-check-target-8jngj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:39.688376 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.688291    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs podName:66062b2c-ba98-4b6a-bc2c-656a6c7c339b nodeName:}" failed. No retries permitted until 2026-09-15 18:50:40.688270615 +0000 UTC m=+2.965039460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs") pod "network-metrics-daemon-9tr5r" (UID: "66062b2c-ba98-4b6a-bc2c-656a6c7c339b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:39.688376 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.688305    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq podName:540b4731-6750-43d4-96b3-759aec9a4eee nodeName:}" failed. No retries permitted until 2026-09-15 18:50:40.688299042 +0000 UTC m=+2.965067876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjsrq" (UniqueName: "kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq") pod "network-check-target-8jngj" (UID: "540b4731-6750-43d4-96b3-759aec9a4eee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:39.789323 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:39.789292    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:39.789485 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.789437    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:39.789521 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:39.789496    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret podName:f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae nodeName:}" failed. No retries permitted until 2026-09-15 18:50:40.789482941 +0000 UTC m=+3.066251776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret") pod "global-pull-secret-syncer-vgmvv" (UID: "f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:40.016036 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.015954    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:40.117294 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.117232    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 18:45:39 +0000 UTC" deadline="2028-02-24 23:07:20.741938175 +0000 UTC"
Sep 15 18:50:40.117294 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.117291    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12652h16m40.624651036s"
Sep 15 18:50:40.295033 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.294752    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:40.295192 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:40.295097    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:50:40.309463 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.309396    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b8g7s" event={"ID":"722d9d5c-0836-40ea-a990-f60b9eff5b5c","Type":"ContainerStarted","Data":"c6dd3f4d0e090d9a9ba5be0b32052075d78cd5f3690c0bdd64a13e621774099a"}
Sep 15 18:50:40.314371 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.314336    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rnhds" event={"ID":"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79","Type":"ContainerStarted","Data":"58359e2e244b900c7112d9b15895967625e14cc5f0f3a32cc1dc6fc0661da839"}
Sep 15 18:50:40.327391 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.327352    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-227.ec2.internal" event={"ID":"527678610c0f9e9b262ca08d3e160568","Type":"ContainerStarted","Data":"df34951f77fa8110e82443927ecefc249d58f53b5301fb11ee262811578bd9be"}
Sep 15 18:50:40.332668 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.332636    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal" event={"ID":"b3d6ec7431a5c04a615ddb019fc77a6f","Type":"ContainerStarted","Data":"0b12dafc4bcb44bfa9dca202b202f0d5fb1961120ea0a11f64bc012cfa114007"}
Sep 15 18:50:40.336358 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.336333    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs" event={"ID":"f4393efb-59bf-4bd4-9cf4-3ab42788041a","Type":"ContainerStarted","Data":"93c6edbfc1e7fc8a23c2333a375a601111e13f83dbcc9e5fad60b02780f6aa3c"}
Sep 15 18:50:40.342835 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.342805    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls" event={"ID":"d019fddd-e527-4f2f-80f7-37ecf6e5ca20","Type":"ContainerStarted","Data":"21082806340f2010efaa27f763774c39efaa21ddd82f4242fec038b807827ef3"}
Sep 15 18:50:40.354666 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.354574    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fnczv" event={"ID":"b3707c81-43dd-4142-9c7b-82b12577c423","Type":"ContainerStarted","Data":"c8eb5348702c530e8c2e6fafaea5f80534807070e7b7a41705c79ba512d895b0"}
Sep 15 18:50:40.368775 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.368733    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvsmg" event={"ID":"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c","Type":"ContainerStarted","Data":"11abfaaa83a99412de1df0b0fb4f7a8e186ae2ba7cf8aaac1931c626c03b575e"}
Sep 15 18:50:40.373497 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:40.373466    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:62c7cc09ff9f9684116e4c6b702d22ac67be29319eb961485e46f9441db87fb0\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-wvsmg" podUID="0dfbf315-3e43-45e4-9a05-4b64c8dbb36c"
Sep 15 18:50:40.379154 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.379125    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-776nl" event={"ID":"45a116a2-6657-4785-8769-056a5708db32","Type":"ContainerStarted","Data":"75e9271e82e516287a7122bde4b7dad390ff0f8c015fb88b296a9cd5ef1b8c26"}
Sep 15 18:50:40.384742 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.384710    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5n5wz" event={"ID":"9fd5a36a-c351-4e64-8b75-abf7e06279a0","Type":"ContainerStarted","Data":"bdbfcc2864c81650a7f6ba4800a7a8ef039d0404b337dff55f222d72d1eabd7b"}
Sep 15 18:50:40.395197 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.395161    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gb7rw" event={"ID":"97ed5cdc-1861-49ab-b028-1dc539ea4ef1","Type":"ContainerStarted","Data":"1d8f372292a1657bc090aeb0d37edef02a9db85941a010d621b9bec06c7ca8c0"}
Sep 15 18:50:40.546587 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.546505    2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:40.696746 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.696707    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs\") pod \"network-metrics-daemon-9tr5r\" (UID: \"66062b2c-ba98-4b6a-bc2c-656a6c7c339b\") " pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:40.696934 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.696798    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjsrq\" (UniqueName: \"kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq\") pod \"network-check-target-8jngj\" (UID: \"540b4731-6750-43d4-96b3-759aec9a4eee\") " pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:40.697001 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:40.696973    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:40.697001 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:40.696996    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:40.697095 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:40.697009    2550 projected.go:196] Error preparing data for projected volume kube-api-access-kjsrq for pod openshift-network-diagnostics/network-check-target-8jngj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:40.697095 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:40.697069    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq podName:540b4731-6750-43d4-96b3-759aec9a4eee nodeName:}" failed. No retries permitted until 2026-09-15 18:50:42.697052359 +0000 UTC m=+4.973821210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjsrq" (UniqueName: "kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq") pod "network-check-target-8jngj" (UID: "540b4731-6750-43d4-96b3-759aec9a4eee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:40.697515 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:40.697494    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:40.697601 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:40.697555    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs podName:66062b2c-ba98-4b6a-bc2c-656a6c7c339b nodeName:}" failed. No retries permitted until 2026-09-15 18:50:42.697539228 +0000 UTC m=+4.974308079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs") pod "network-metrics-daemon-9tr5r" (UID: "66062b2c-ba98-4b6a-bc2c-656a6c7c339b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:40.797835 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:40.797180    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:40.797835 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:40.797358    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:40.797835 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:40.797424    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret podName:f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae nodeName:}" failed. No retries permitted until 2026-09-15 18:50:42.797406701 +0000 UTC m=+5.074175538 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret") pod "global-pull-secret-syncer-vgmvv" (UID: "f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:41.118302 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:41.118209    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 18:45:39 +0000 UTC" deadline="2028-02-19 19:36:43.349829659 +0000 UTC"
Sep 15 18:50:41.118302 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:41.118255    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12528h46m2.231581532s"
Sep 15 18:50:41.289055 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:41.289007    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:41.289233 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:41.289150    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:50:41.289635 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:41.289615    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:41.290004 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:41.289725    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:50:42.174623 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:42.174378    2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:42.293525 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:42.290450    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:42.293525 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:42.290620    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:50:42.713685 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:42.713643    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjsrq\" (UniqueName: \"kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq\") pod \"network-check-target-8jngj\" (UID: \"540b4731-6750-43d4-96b3-759aec9a4eee\") " pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:42.713871 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:42.713730    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs\") pod \"network-metrics-daemon-9tr5r\" (UID: \"66062b2c-ba98-4b6a-bc2c-656a6c7c339b\") " pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:42.713947 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:42.713872    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:42.713947 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:42.713938    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs podName:66062b2c-ba98-4b6a-bc2c-656a6c7c339b nodeName:}" failed. No retries permitted until 2026-09-15 18:50:46.713919356 +0000 UTC m=+8.990688206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs") pod "network-metrics-daemon-9tr5r" (UID: "66062b2c-ba98-4b6a-bc2c-656a6c7c339b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:42.714507 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:42.714486    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:42.714507 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:42.714510    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:42.714682 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:42.714524    2550 projected.go:196] Error preparing data for projected volume kube-api-access-kjsrq for pod openshift-network-diagnostics/network-check-target-8jngj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:42.714682 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:42.714571    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq podName:540b4731-6750-43d4-96b3-759aec9a4eee nodeName:}" failed. No retries permitted until 2026-09-15 18:50:46.714555553 +0000 UTC m=+8.991324387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjsrq" (UniqueName: "kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq") pod "network-check-target-8jngj" (UID: "540b4731-6750-43d4-96b3-759aec9a4eee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:42.814364 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:42.814308    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:42.814549 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:42.814459    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:42.814549 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:42.814534    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret podName:f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae nodeName:}" failed. No retries permitted until 2026-09-15 18:50:46.814515267 +0000 UTC m=+9.091284116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret") pod "global-pull-secret-syncer-vgmvv" (UID: "f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:43.290164 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:43.289519    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:43.290164 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:43.289630    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:50:43.290164 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:43.289707    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:43.290164 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:43.289757    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:50:44.289708 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:44.289676    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:44.289906 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:44.289832    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:50:45.289498 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:45.289453    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:45.289926 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:45.289455    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:45.289926 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:45.289591    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:50:45.289926 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:45.289688    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:50:46.292579 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:46.292552    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:46.293020 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:46.292663    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:50:46.747140 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:46.747051    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs\") pod \"network-metrics-daemon-9tr5r\" (UID: \"66062b2c-ba98-4b6a-bc2c-656a6c7c339b\") " pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:46.747306 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:46.747156    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjsrq\" (UniqueName: \"kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq\") pod \"network-check-target-8jngj\" (UID: \"540b4731-6750-43d4-96b3-759aec9a4eee\") " pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:46.747397 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:46.747374    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:46.747461 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:46.747399    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:46.747461 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:46.747414    2550 projected.go:196] Error preparing data for projected volume kube-api-access-kjsrq for pod openshift-network-diagnostics/network-check-target-8jngj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:46.747559 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:46.747475    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq podName:540b4731-6750-43d4-96b3-759aec9a4eee nodeName:}" failed. No retries permitted until 2026-09-15 18:50:54.747456105 +0000 UTC m=+17.024224952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjsrq" (UniqueName: "kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq") pod "network-check-target-8jngj" (UID: "540b4731-6750-43d4-96b3-759aec9a4eee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:46.747923 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:46.747898    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:46.748006 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:46.747962    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs podName:66062b2c-ba98-4b6a-bc2c-656a6c7c339b nodeName:}" failed. No retries permitted until 2026-09-15 18:50:54.747944541 +0000 UTC m=+17.024713380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs") pod "network-metrics-daemon-9tr5r" (UID: "66062b2c-ba98-4b6a-bc2c-656a6c7c339b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:46.847928 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:46.847886    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:46.848103 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:46.848021    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:46.848103 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:46.848087    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret podName:f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae nodeName:}" failed. No retries permitted until 2026-09-15 18:50:54.848071573 +0000 UTC m=+17.124840420 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret") pod "global-pull-secret-syncer-vgmvv" (UID: "f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:47.289643 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:47.289018    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:47.289643 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:47.289068    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:47.289643 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:47.289172    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:50:47.289643 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:47.289262    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:50:48.290702 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:48.290619    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:48.291167 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:48.290771    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:50:49.289677 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:49.289641    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:49.289841 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:49.289641    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:49.289841 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:49.289768    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:50:49.289841 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:49.289832    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:50:50.290124 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:50.290040    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:50.290676 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:50.290181    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:50:51.289187 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:51.289150    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:51.289400 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:51.289290    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:50:51.289400 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:51.289385    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:51.289512 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:51.289452    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:50:52.289408 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:52.289375    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:52.289890 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:52.289514    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:50:53.290101 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:53.290070    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:53.290594 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:53.290178    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:53.290594 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:53.290331    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:50:53.290594 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:53.290404    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:50:54.289271 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:54.289238    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:54.289458 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:54.289399    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:50:54.806504 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:54.806459    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs\") pod \"network-metrics-daemon-9tr5r\" (UID: \"66062b2c-ba98-4b6a-bc2c-656a6c7c339b\") " pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:54.806979 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:54.806535    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjsrq\" (UniqueName: \"kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq\") pod \"network-check-target-8jngj\" (UID: \"540b4731-6750-43d4-96b3-759aec9a4eee\") " pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:54.806979 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:54.806603    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:54.806979 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:54.806644    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:54.806979 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:54.806662    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:54.806979 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:54.806676    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs podName:66062b2c-ba98-4b6a-bc2c-656a6c7c339b nodeName:}" failed. No retries permitted until 2026-09-15 18:51:10.806656262 +0000 UTC m=+33.083425117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs") pod "network-metrics-daemon-9tr5r" (UID: "66062b2c-ba98-4b6a-bc2c-656a6c7c339b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:54.806979 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:54.806676    2550 projected.go:196] Error preparing data for projected volume kube-api-access-kjsrq for pod openshift-network-diagnostics/network-check-target-8jngj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:54.806979 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:54.806709    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq podName:540b4731-6750-43d4-96b3-759aec9a4eee nodeName:}" failed. No retries permitted until 2026-09-15 18:51:10.806702985 +0000 UTC m=+33.083471848 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjsrq" (UniqueName: "kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq") pod "network-check-target-8jngj" (UID: "540b4731-6750-43d4-96b3-759aec9a4eee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:54.907545 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:54.907505    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:54.907717 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:54.907668    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:54.907775 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:54.907746    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret podName:f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae nodeName:}" failed. No retries permitted until 2026-09-15 18:51:10.907726786 +0000 UTC m=+33.184495627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret") pod "global-pull-secret-syncer-vgmvv" (UID: "f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:55.289489 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:55.289402    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:55.289489 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:55.289448    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:55.289709 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:55.289537    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:50:55.289709 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:55.289650    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:50:56.289919 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:56.289883    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:56.290354 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:56.290039    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:50:57.289520 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:57.289488    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:57.289651 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:57.289495    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:57.289651 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:57.289585    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:50:57.289746 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:57.289664    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:50:58.291260 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.290865    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:50:58.292080 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:58.291301    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:50:58.432889 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.432846    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-227.ec2.internal" event={"ID":"527678610c0f9e9b262ca08d3e160568","Type":"ContainerStarted","Data":"3a6a81881c556c70ff42a8b1cd76b99660097b58111dea87d1c846beee1d720e"}
Sep 15 18:50:58.435077 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.435052    2550 generic.go:356] "Generic (PLEG): container finished" podID="b3d6ec7431a5c04a615ddb019fc77a6f" containerID="7b16780937fde2458eb0bc938fd428f3c62ab0c3d439134b947d1be1808e6774" exitCode=0
Sep 15 18:50:58.435223 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.435141    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal" event={"ID":"b3d6ec7431a5c04a615ddb019fc77a6f","Type":"ContainerDied","Data":"7b16780937fde2458eb0bc938fd428f3c62ab0c3d439134b947d1be1808e6774"}
Sep 15 18:50:58.437143 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.437057    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs" event={"ID":"f4393efb-59bf-4bd4-9cf4-3ab42788041a","Type":"ContainerStarted","Data":"e1126ac575b4aa6b451ba27d94458e042bc52e1590dba1085935278d13ca1858"}
Sep 15 18:50:58.440638 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.440612    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls" event={"ID":"d019fddd-e527-4f2f-80f7-37ecf6e5ca20","Type":"ContainerStarted","Data":"e6193579b3eb82d476d9ba7e466ffc201bd59bbac715707658fba2af732fe91d"}
Sep 15 18:50:58.440773 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.440676    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls" event={"ID":"d019fddd-e527-4f2f-80f7-37ecf6e5ca20","Type":"ContainerStarted","Data":"3bc80d499ad9e442b2253c25cd8c3d81f100eb39f42afd0346c9b52cdcad34b6"}
Sep 15 18:50:58.440773 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.440713    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls" event={"ID":"d019fddd-e527-4f2f-80f7-37ecf6e5ca20","Type":"ContainerStarted","Data":"48b1906d649147dff134ceb9da4f0f23753eb5b8e3630bf1aaa3b0180a52a240"}
Sep 15 18:50:58.440773 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.440745    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls" event={"ID":"d019fddd-e527-4f2f-80f7-37ecf6e5ca20","Type":"ContainerStarted","Data":"37eea24fb40449d5dac2c5ff1f124578921c8c62fc382de374453f7480cce9bb"}
Sep 15 18:50:58.440899 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.440781    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls" event={"ID":"d019fddd-e527-4f2f-80f7-37ecf6e5ca20","Type":"ContainerStarted","Data":"6282bc32d91c7ca30c7e41e32e7d14c4a6a8d09a29c085aa08504c8ac90c3de1"}
Sep 15 18:50:58.440899 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.440815    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls" event={"ID":"d019fddd-e527-4f2f-80f7-37ecf6e5ca20","Type":"ContainerStarted","Data":"3d91e37f9a81530b5f179482f0fffec578fdc877ae8283b01fb34bd9877188e2"}
Sep 15 18:50:58.442717 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.442337    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fnczv" event={"ID":"b3707c81-43dd-4142-9c7b-82b12577c423","Type":"ContainerStarted","Data":"da00b58a47da1dad06542399ec40c2273321d9bec4e771a5c9af692fb4f6b4cf"}
Sep 15 18:50:58.443922 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.443897    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-776nl" event={"ID":"45a116a2-6657-4785-8769-056a5708db32","Type":"ContainerStarted","Data":"0608fd1d1de6cb22f01039750563341267b8151fd2109b120350783ab1afcdfc"}
Sep 15 18:50:58.445556 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.445497    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5n5wz" event={"ID":"9fd5a36a-c351-4e64-8b75-abf7e06279a0","Type":"ContainerStarted","Data":"9844e350bdb1f56b581f66ea281404cb73bd6d8ce62a4f04db62bd10454b9128"}
Sep 15 18:50:58.447074 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.446991    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-227.ec2.internal" podStartSLOduration=20.446976492 podStartE2EDuration="20.446976492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:50:58.446332558 +0000 UTC m=+20.723101415" watchObservedRunningTime="2026-09-15 18:50:58.446976492 +0000 UTC m=+20.723745349"
Sep 15 18:50:58.447205 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.447184    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gb7rw" event={"ID":"97ed5cdc-1861-49ab-b028-1dc539ea4ef1","Type":"ContainerStarted","Data":"af812ca110fd5141f62e9824f45cb30765823c846379979f5c4076b5959b1799"}
Sep 15 18:50:58.449166 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.448807    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rnhds" event={"ID":"f07817f7-e8cf-4ba0-a211-9e3d85ee4f79","Type":"ContainerStarted","Data":"0ee2c02413f50b5252619c3ea6bbdba4e187b386a122d0d1e782692e89514a57"}
Sep 15 18:50:58.459619 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.459578    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-776nl" podStartSLOduration=2.628982126 podStartE2EDuration="20.459561532s" totalImagesPullingTime="17.830579406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:50:58.459384612 +0000 UTC m=+20.736153470" watchObservedRunningTime="2026-09-15 18:50:58.459561532 +0000 UTC m=+20.736330389"
Sep 15 18:50:58.485889 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.485839    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fnczv" podStartSLOduration=2.545549862 podStartE2EDuration="20.485824212s" totalImagesPullingTime="17.94027435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:50:58.485405268 +0000 UTC m=+20.762174125" watchObservedRunningTime="2026-09-15 18:50:58.485824212 +0000 UTC m=+20.762593068"
Sep 15 18:50:58.501713 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.501668    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5n5wz" podStartSLOduration=2.665372923 podStartE2EDuration="20.501652921s" totalImagesPullingTime="17.836279998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:50:58.501184575 +0000 UTC m=+20.777953434" watchObservedRunningTime="2026-09-15 18:50:58.501652921 +0000 UTC m=+20.778421774"
Sep 15 18:50:58.517769 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.517719    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rnhds" podStartSLOduration=2.461161106 podStartE2EDuration="20.517702597s" totalImagesPullingTime="18.056541491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:50:58.517439511 +0000 UTC m=+20.794208369" watchObservedRunningTime="2026-09-15 18:50:58.517702597 +0000 UTC m=+20.794471452"
Sep 15 18:50:58.531936 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:58.531888    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gb7rw" podStartSLOduration=2.664817567 podStartE2EDuration="20.531872844s" totalImagesPullingTime="17.867055277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:50:58.531558663 +0000 UTC m=+20.808327519" watchObservedRunningTime="2026-09-15 18:50:58.531872844 +0000 UTC m=+20.808641701"
Sep 15 18:50:59.289858 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:59.289778    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:50:59.290010 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:59.289778    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:50:59.290010 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:59.289906    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:50:59.290010 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:50:59.289968    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:50:59.452612 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:59.452581    2550 generic.go:356] "Generic (PLEG): container finished" podID="0dfbf315-3e43-45e4-9a05-4b64c8dbb36c" containerID="8cae59a05dd6a344fe3de17ae53661acac7d8191544c740936168b7aeb28a799" exitCode=0
Sep 15 18:50:59.453382 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:59.452654    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvsmg" event={"ID":"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c","Type":"ContainerDied","Data":"8cae59a05dd6a344fe3de17ae53661acac7d8191544c740936168b7aeb28a799"}
Sep 15 18:50:59.454302 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:59.454242    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b8g7s" event={"ID":"722d9d5c-0836-40ea-a990-f60b9eff5b5c","Type":"ContainerStarted","Data":"1aefca3a17e8abae993007006f05c6d3877381da2ec00326030b5656bd6d7bda"}
Sep 15 18:50:59.456264 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:59.456195    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal" event={"ID":"b3d6ec7431a5c04a615ddb019fc77a6f","Type":"ContainerStarted","Data":"f6c1ba469b605ae3560c9c2afe349cc7d80fc9ff168197ea51cec316529a5366"}
Sep 15 18:50:59.485793 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:59.485555    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b8g7s" podStartSLOduration=3.567479765 podStartE2EDuration="21.485537085s" totalImagesPullingTime="17.91805732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:50:59.48511923 +0000 UTC m=+21.761888087" watchObservedRunningTime="2026-09-15 18:50:59.485537085 +0000 UTC m=+21.762305942"
Sep 15 18:50:59.498330 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:59.498267    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-227.ec2.internal" podStartSLOduration=21.498252448 podStartE2EDuration="21.498252448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:50:59.498111558 +0000 UTC m=+21.774880413" watchObservedRunningTime="2026-09-15 18:50:59.498252448 +0000 UTC m=+21.775021304"
Sep 15 18:50:59.556109 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:59.556079    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fnczv"
Sep 15 18:50:59.556613 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:59.556598    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fnczv"
Sep 15 18:50:59.593931 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:50:59.593909    2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 18:51:00.154579 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:00.154479    2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T18:50:59.593927182Z","UUID":"3d80ace6-e66b-4c48-8c28-903d83ee866f","Handler":null,"Name":"","Endpoint":""}
Sep 15 18:51:00.156033 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:00.156000    2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 18:51:00.156163 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:00.156046    2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 18:51:00.290157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:00.290127    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:51:00.290521 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:00.290284    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:51:00.461708 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:00.461665    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs" event={"ID":"f4393efb-59bf-4bd4-9cf4-3ab42788041a","Type":"ContainerStarted","Data":"aa8e850e5e2f5ca7d76f1576b857867d29420702bb21c931d0f6ab7e73b743a7"}
Sep 15 18:51:00.465963 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:00.465931    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls" event={"ID":"d019fddd-e527-4f2f-80f7-37ecf6e5ca20","Type":"ContainerStarted","Data":"5943bd99c64e4effde6f14796cb51e3f2eeff9be9b806edd45985e577a2e99ec"}
Sep 15 18:51:00.466611 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:00.466579    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fnczv"
Sep 15 18:51:00.469413 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:00.467399    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fnczv"
Sep 15 18:51:01.290043 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:01.290012    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:51:01.290043 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:01.290057    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:51:01.290285 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:01.290151    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:51:01.290285 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:01.290232    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:51:01.472326 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:01.472235    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs" event={"ID":"f4393efb-59bf-4bd4-9cf4-3ab42788041a","Type":"ContainerStarted","Data":"832a8b8faebed46b4222d53c8ef9b76c3cafe758125b1536a2c51aa9732c4e2a"}
Sep 15 18:51:01.488971 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:01.488896    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9bgs" podStartSLOduration=3.137179377 podStartE2EDuration="23.488879778s" totalImagesPullingTime="20.351700401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:01.488013164 +0000 UTC m=+23.764782087" watchObservedRunningTime="2026-09-15 18:51:01.488879778 +0000 UTC m=+23.765648636"
Sep 15 18:51:02.289598 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:02.289563    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:51:02.289787 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:02.289706    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:51:03.289743 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:03.289706    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:51:03.290202 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:03.289706    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:51:03.290202 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:03.289828    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:51:03.290202 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:03.289918    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:51:04.289433 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:04.289187    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:51:04.289653 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:04.289469    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:51:04.482558 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:04.482524    2550 generic.go:356] "Generic (PLEG): container finished" podID="0dfbf315-3e43-45e4-9a05-4b64c8dbb36c" containerID="db14bad2b17d413af507238bc2565c3944809d93b6d9de6fa4cc8cf19ba73fad" exitCode=0
Sep 15 18:51:04.483078 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:04.482601    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvsmg" event={"ID":"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c","Type":"ContainerDied","Data":"db14bad2b17d413af507238bc2565c3944809d93b6d9de6fa4cc8cf19ba73fad"}
Sep 15 18:51:04.488377 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:04.488347    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls" event={"ID":"d019fddd-e527-4f2f-80f7-37ecf6e5ca20","Type":"ContainerStarted","Data":"27bba45cdd2fe9cda4addbbd178539dde493dbd3847a7972ec45e50136b5d4c0"}
Sep 15 18:51:04.488554 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:04.488536    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:51:04.488710 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:04.488591    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:51:04.504683 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:04.504661    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:51:04.526896 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:04.526856    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls" podStartSLOduration=8.585755984 podStartE2EDuration="26.526824673s" totalImagesPullingTime="17.941068689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:04.526223319 +0000 UTC m=+26.802992174" watchObservedRunningTime="2026-09-15 18:51:04.526824673 +0000 UTC m=+26.803593528"
Sep 15 18:51:05.289881 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:05.289854    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:51:05.289996 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:05.289854    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:51:05.289996 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:05.289965    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:51:05.290065 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:05.290017    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:51:05.438951 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:05.438740    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9tr5r"]
Sep 15 18:51:05.439105 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:05.439044    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:51:05.439152 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:05.439132    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:51:05.441299 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:05.441260    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vgmvv"]
Sep 15 18:51:05.441977 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:05.441953    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8jngj"]
Sep 15 18:51:05.492597 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:05.492525    2550 generic.go:356] "Generic (PLEG): container finished" podID="0dfbf315-3e43-45e4-9a05-4b64c8dbb36c" containerID="9cf62cee7aeb9222ff5950e5449dd0f2ebf16cf94957323997d593bb42b80bcf" exitCode=0
Sep 15 18:51:05.493231 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:05.492615    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvsmg" event={"ID":"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c","Type":"ContainerDied","Data":"9cf62cee7aeb9222ff5950e5449dd0f2ebf16cf94957323997d593bb42b80bcf"}
Sep 15 18:51:05.493231 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:05.492660    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:51:05.493231 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:05.492628    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:51:05.493231 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:05.492776    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:51:05.493231 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:05.492861    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:51:05.493231 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:05.492937    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:51:05.508441 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:05.508421    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:51:06.497436 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:06.497349    2550 generic.go:356] "Generic (PLEG): container finished" podID="0dfbf315-3e43-45e4-9a05-4b64c8dbb36c" containerID="39dc1e00fb3b87e07c98e321af72868ef27e5f94a0c132d0088d03f47f2bbfa6" exitCode=0
Sep 15 18:51:06.497436 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:06.497401    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvsmg" event={"ID":"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c","Type":"ContainerDied","Data":"39dc1e00fb3b87e07c98e321af72868ef27e5f94a0c132d0088d03f47f2bbfa6"}
Sep 15 18:51:07.289457 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:07.289420    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:51:07.289457 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:07.289447    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:51:07.289654 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:07.289576    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:51:07.289654 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:07.289597    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:51:07.289654 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:07.289648    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:51:07.289787 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:07.289712    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:51:09.289153 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:09.288990    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:51:09.289153 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:09.288991    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:51:09.289153 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:09.289101    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jngj" podUID="540b4731-6750-43d4-96b3-759aec9a4eee"
Sep 15 18:51:09.289780 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:09.289188    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tr5r" podUID="66062b2c-ba98-4b6a-bc2c-656a6c7c339b"
Sep 15 18:51:09.289780 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:09.288991    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:51:09.289780 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:09.289270    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vgmvv" podUID="f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae"
Sep 15 18:51:10.463502 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.463466    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-227.ec2.internal" event="NodeReady"
Sep 15 18:51:10.463940 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.463602    2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 18:51:10.515283 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.515204    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bq2c6"]
Sep 15 18:51:10.556193 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.556155    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lvxks"]
Sep 15 18:51:10.556392 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.556277    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:10.558424 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.558392    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 18:51:10.558568 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.558436    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 18:51:10.558749 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.558722    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcddk\""
Sep 15 18:51:10.583850 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.583820    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7dc58f8f87-zjl2r"]
Sep 15 18:51:10.584019 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.583921    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lvxks"
Sep 15 18:51:10.586331 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.586293    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 18:51:10.586462 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.586389    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6fg9f\""
Sep 15 18:51:10.586462 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.586414    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 18:51:10.586659 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.586637    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 18:51:10.600145 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.600119    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bq2c6"]
Sep 15 18:51:10.600284 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.600199    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lvxks"]
Sep 15 18:51:10.600284 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.600221    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dc58f8f87-zjl2r"]
Sep 15 18:51:10.600408 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.600299    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.602921 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.602901    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 18:51:10.603076 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.602904    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nk7cf\""
Sep 15 18:51:10.603076 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.602944    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 18:51:10.603076 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.602983    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 18:51:10.607209 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.607189    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 18:51:10.626572 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.626535    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpzd9\" (UniqueName: \"kubernetes.io/projected/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-kube-api-access-wpzd9\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:10.626718 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.626631    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:10.626718 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.626686    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-config-volume\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:10.626817 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.626717    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-tmp-dir\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:10.727832 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.727792    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:10.727832 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.727834    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dada676b-2a65-4f7c-b43f-61969e29ad85-image-registry-private-configuration\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.728089 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.727886    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dada676b-2a65-4f7c-b43f-61969e29ad85-ca-trust-extracted\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.728089 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.727912    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-config-volume\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:10.728089 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:10.727980    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:10.728089 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.727991    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-tmp-dir\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:10.728089 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.728030    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rg7n\" (UniqueName: \"kubernetes.io/projected/8216bdee-a7c9-4523-8732-ba0dbba092fd-kube-api-access-7rg7n\") pod \"ingress-canary-lvxks\" (UID: \"8216bdee-a7c9-4523-8732-ba0dbba092fd\") " pod="openshift-ingress-canary/ingress-canary-lvxks"
Sep 15 18:51:10.728089 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:10.728058    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls podName:69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:11.228034275 +0000 UTC m=+33.504803123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls") pod "dns-default-bq2c6" (UID: "69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:10.728414 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.728148    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dada676b-2a65-4f7c-b43f-61969e29ad85-trusted-ca\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.728414 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.728176    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klbk8\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-kube-api-access-klbk8\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.728414 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.728204    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpzd9\" (UniqueName: \"kubernetes.io/projected/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-kube-api-access-wpzd9\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:10.728414 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.728242    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dada676b-2a65-4f7c-b43f-61969e29ad85-installation-pull-secrets\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.728414 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.728283    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8216bdee-a7c9-4523-8732-ba0dbba092fd-cert\") pod \"ingress-canary-lvxks\" (UID: \"8216bdee-a7c9-4523-8732-ba0dbba092fd\") " pod="openshift-ingress-canary/ingress-canary-lvxks"
Sep 15 18:51:10.728414 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.728346    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.728414 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.728377    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-bound-sa-token\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.728710 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.728446    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-tmp-dir\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:10.728710 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.728473    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-certificates\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.728710 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.728619    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-config-volume\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:10.740206 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.740182    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpzd9\" (UniqueName: \"kubernetes.io/projected/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-kube-api-access-wpzd9\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:10.829370 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.829254    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-certificates\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.829518 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.829378    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dada676b-2a65-4f7c-b43f-61969e29ad85-image-registry-private-configuration\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.829518 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.829414    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs\") pod \"network-metrics-daemon-9tr5r\" (UID: \"66062b2c-ba98-4b6a-bc2c-656a6c7c339b\") " pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:51:10.829518 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.829449    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dada676b-2a65-4f7c-b43f-61969e29ad85-ca-trust-extracted\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.829518 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.829489    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rg7n\" (UniqueName: \"kubernetes.io/projected/8216bdee-a7c9-4523-8732-ba0dbba092fd-kube-api-access-7rg7n\") pod \"ingress-canary-lvxks\" (UID: \"8216bdee-a7c9-4523-8732-ba0dbba092fd\") " pod="openshift-ingress-canary/ingress-canary-lvxks"
Sep 15 18:51:10.829518 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.829516    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dada676b-2a65-4f7c-b43f-61969e29ad85-trusted-ca\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.829837 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.829542    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klbk8\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-kube-api-access-klbk8\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.829837 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:10.829561    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:10.829837 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:10.829649    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs podName:66062b2c-ba98-4b6a-bc2c-656a6c7c339b nodeName:}" failed. No retries permitted until 2026-09-15 18:51:42.829629664 +0000 UTC m=+65.106398499 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs") pod "network-metrics-daemon-9tr5r" (UID: "66062b2c-ba98-4b6a-bc2c-656a6c7c339b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:10.829837 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.829572    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dada676b-2a65-4f7c-b43f-61969e29ad85-installation-pull-secrets\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.830035 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.829876    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8216bdee-a7c9-4523-8732-ba0dbba092fd-cert\") pod \"ingress-canary-lvxks\" (UID: \"8216bdee-a7c9-4523-8732-ba0dbba092fd\") " pod="openshift-ingress-canary/ingress-canary-lvxks"
Sep 15 18:51:10.830035 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.829918    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.830035 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.829930    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dada676b-2a65-4f7c-b43f-61969e29ad85-ca-trust-extracted\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.830035 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.829966    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-certificates\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.830035 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.829945    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-bound-sa-token\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.830035 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.830016    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjsrq\" (UniqueName: \"kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq\") pod \"network-check-target-8jngj\" (UID: \"540b4731-6750-43d4-96b3-759aec9a4eee\") " pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:51:10.830333 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:10.830120    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:51:10.830333 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:10.830141    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:51:10.830333 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:10.830141    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:10.830333 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:10.830153    2550 projected.go:196] Error preparing data for projected volume kube-api-access-kjsrq for pod openshift-network-diagnostics/network-check-target-8jngj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:10.830333 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:10.830155    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dc58f8f87-zjl2r: secret "image-registry-tls" not found
Sep 15 18:51:10.830333 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:10.830221    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls podName:dada676b-2a65-4f7c-b43f-61969e29ad85 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:11.330203708 +0000 UTC m=+33.606972543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls") pod "image-registry-7dc58f8f87-zjl2r" (UID: "dada676b-2a65-4f7c-b43f-61969e29ad85") : secret "image-registry-tls" not found
Sep 15 18:51:10.830333 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:10.830240    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq podName:540b4731-6750-43d4-96b3-759aec9a4eee nodeName:}" failed. No retries permitted until 2026-09-15 18:51:42.830229652 +0000 UTC m=+65.106998491 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjsrq" (UniqueName: "kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq") pod "network-check-target-8jngj" (UID: "540b4731-6750-43d4-96b3-759aec9a4eee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:10.830644 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.830431    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dada676b-2a65-4f7c-b43f-61969e29ad85-trusted-ca\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.832813 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.832790    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dada676b-2a65-4f7c-b43f-61969e29ad85-installation-pull-secrets\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.832937 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.832818    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dada676b-2a65-4f7c-b43f-61969e29ad85-image-registry-private-configuration\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.832937 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.832858    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8216bdee-a7c9-4523-8732-ba0dbba092fd-cert\") pod \"ingress-canary-lvxks\" (UID: \"8216bdee-a7c9-4523-8732-ba0dbba092fd\") " pod="openshift-ingress-canary/ingress-canary-lvxks"
Sep 15 18:51:10.837462 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.837440    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rg7n\" (UniqueName: \"kubernetes.io/projected/8216bdee-a7c9-4523-8732-ba0dbba092fd-kube-api-access-7rg7n\") pod \"ingress-canary-lvxks\" (UID: \"8216bdee-a7c9-4523-8732-ba0dbba092fd\") " pod="openshift-ingress-canary/ingress-canary-lvxks"
Sep 15 18:51:10.837590 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.837464    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-bound-sa-token\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.837590 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.837463    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klbk8\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-kube-api-access-klbk8\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:10.894212 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.894176    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lvxks"
Sep 15 18:51:10.931338 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:10.931292    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:51:10.931517 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:10.931424    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:10.931517 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:10.931506    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret podName:f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae nodeName:}" failed. No retries permitted until 2026-09-15 18:51:42.931487506 +0000 UTC m=+65.208256359 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret") pod "global-pull-secret-syncer-vgmvv" (UID: "f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:11.233500 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:11.233420    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:11.233651 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:11.233599    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:11.233712 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:11.233681    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls podName:69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:12.233661243 +0000 UTC m=+34.510430094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls") pod "dns-default-bq2c6" (UID: "69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:11.289467 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:11.289426    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:51:11.289639 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:11.289426    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:51:11.289639 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:11.289445    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:51:11.291872 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:11.291848    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 18:51:11.291872 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:11.291865    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4h8bk\""
Sep 15 18:51:11.292043 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:11.291916    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 18:51:11.292043 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:11.291854    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gnvqj\""
Sep 15 18:51:11.292274 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:11.292257    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 18:51:11.292416 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:11.292391    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 18:51:11.334200 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:11.334175    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:11.334374 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:11.334330    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:11.334374 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:11.334344    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dc58f8f87-zjl2r: secret "image-registry-tls" not found
Sep 15 18:51:11.334453 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:11.334400    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls podName:dada676b-2a65-4f7c-b43f-61969e29ad85 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:12.334383773 +0000 UTC m=+34.611152619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls") pod "image-registry-7dc58f8f87-zjl2r" (UID: "dada676b-2a65-4f7c-b43f-61969e29ad85") : secret "image-registry-tls" not found
Sep 15 18:51:11.644286 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:11.643999    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lvxks"]
Sep 15 18:51:11.649076 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:51:11.649042    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8216bdee_a7c9_4523_8732_ba0dbba092fd.slice/crio-9fd58e4173a7ae8a0ab74790399159a0000e09949f218194212ff610659cb8e4 WatchSource:0}: Error finding container 9fd58e4173a7ae8a0ab74790399159a0000e09949f218194212ff610659cb8e4: Status 404 returned error can't find the container with id 9fd58e4173a7ae8a0ab74790399159a0000e09949f218194212ff610659cb8e4
Sep 15 18:51:12.241970 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:12.241883    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:12.242119 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:12.241998    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:12.242119 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:12.242052    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls podName:69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:14.242037501 +0000 UTC m=+36.518806348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls") pod "dns-default-bq2c6" (UID: "69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:12.342679 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:12.342639    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:12.342857 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:12.342796    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:12.342857 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:12.342813    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dc58f8f87-zjl2r: secret "image-registry-tls" not found
Sep 15 18:51:12.342975 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:12.342872    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls podName:dada676b-2a65-4f7c-b43f-61969e29ad85 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:14.342854135 +0000 UTC m=+36.619622992 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls") pod "image-registry-7dc58f8f87-zjl2r" (UID: "dada676b-2a65-4f7c-b43f-61969e29ad85") : secret "image-registry-tls" not found
Sep 15 18:51:12.512176 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:12.512083    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lvxks" event={"ID":"8216bdee-a7c9-4523-8732-ba0dbba092fd","Type":"ContainerStarted","Data":"9fd58e4173a7ae8a0ab74790399159a0000e09949f218194212ff610659cb8e4"}
Sep 15 18:51:12.515236 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:12.515203    2550 generic.go:356] "Generic (PLEG): container finished" podID="0dfbf315-3e43-45e4-9a05-4b64c8dbb36c" containerID="7e1491053b7244308b2c670f3ceb51c975a81249d5b4c15f4c60ab77dcfffc90" exitCode=0
Sep 15 18:51:12.515406 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:12.515277    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvsmg" event={"ID":"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c","Type":"ContainerDied","Data":"7e1491053b7244308b2c670f3ceb51c975a81249d5b4c15f4c60ab77dcfffc90"}
Sep 15 18:51:13.522235 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:13.522201    2550 generic.go:356] "Generic (PLEG): container finished" podID="0dfbf315-3e43-45e4-9a05-4b64c8dbb36c" containerID="bd9d108cabef57449f2490d39e323b702b8db4e2ff70e5ff8f3dcfc620c25beb" exitCode=0
Sep 15 18:51:13.523053 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:13.522271    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvsmg" event={"ID":"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c","Type":"ContainerDied","Data":"bd9d108cabef57449f2490d39e323b702b8db4e2ff70e5ff8f3dcfc620c25beb"}
Sep 15 18:51:14.257163 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:14.257135    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:14.257303 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:14.257282    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:14.257454 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:14.257442    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls podName:69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:18.257423246 +0000 UTC m=+40.534192079 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls") pod "dns-default-bq2c6" (UID: "69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:14.357541 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:14.357511    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:14.357699 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:14.357654    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:14.357699 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:14.357673    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dc58f8f87-zjl2r: secret "image-registry-tls" not found
Sep 15 18:51:14.357770 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:14.357727    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls podName:dada676b-2a65-4f7c-b43f-61969e29ad85 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:18.357711017 +0000 UTC m=+40.634479856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls") pod "image-registry-7dc58f8f87-zjl2r" (UID: "dada676b-2a65-4f7c-b43f-61969e29ad85") : secret "image-registry-tls" not found
Sep 15 18:51:14.528593 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:14.528502    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvsmg" event={"ID":"0dfbf315-3e43-45e4-9a05-4b64c8dbb36c","Type":"ContainerStarted","Data":"5d4baff7eb860ef15c240ed59aaf9617ae293fcd40da7a1a5ef8e7f145df0dab"}
Sep 15 18:51:14.530115 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:14.530088    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lvxks" event={"ID":"8216bdee-a7c9-4523-8732-ba0dbba092fd","Type":"ContainerStarted","Data":"b426b7dc1ee6684cde7c051855ae010841cb02fb6456b7291e8c5a54572bf21a"}
Sep 15 18:51:14.547759 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:14.547692    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wvsmg" podStartSLOduration=4.440157193 podStartE2EDuration="36.547674446s" totalImagesPullingTime="32.107517253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-15 18:51:14.546216358 +0000 UTC m=+36.822985218" watchObservedRunningTime="2026-09-15 18:51:14.547674446 +0000 UTC m=+36.824443302"
Sep 15 18:51:14.561198 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:14.561143    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lvxks" podStartSLOduration=1.8529692450000002 podStartE2EDuration="4.561121764s" totalImagesPullingTime="2.708152519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:14.560462165 +0000 UTC m=+36.837231022" watchObservedRunningTime="2026-09-15 18:51:14.561121764 +0000 UTC m=+36.837890622"
Sep 15 18:51:16.847010 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:16.846978    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qz4bv"]
Sep 15 18:51:16.878100 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:16.878068    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qz4bv"]
Sep 15 18:51:16.878234 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:16.878192    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:16.880745 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:16.880723    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 18:51:16.880745 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:16.880733    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 18:51:16.880989 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:16.880974    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 18:51:16.881596 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:16.881579    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 18:51:16.881661 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:16.881648    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pj5l2\""
Sep 15 18:51:16.976946 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:16.976916    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8ca2cbaa-a242-425a-8886-c327cee392b2-crio-socket\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:16.976946 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:16.976952    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czsv7\" (UniqueName: \"kubernetes.io/projected/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-api-access-czsv7\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:16.977116 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:16.977055    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:16.977116 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:16.977091    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ca2cbaa-a242-425a-8886-c327cee392b2-data-volume\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:16.977208 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:16.977127    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:17.077611 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.077575    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8ca2cbaa-a242-425a-8886-c327cee392b2-crio-socket\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:17.077737 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.077618    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czsv7\" (UniqueName: \"kubernetes.io/projected/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-api-access-czsv7\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:17.077803 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.077740    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:17.077803 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.077764    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ca2cbaa-a242-425a-8886-c327cee392b2-data-volume\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:17.077803 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.077791    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:17.077928 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.077830    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8ca2cbaa-a242-425a-8886-c327cee392b2-crio-socket\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:17.077928 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:17.077896    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:51:17.078021 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:17.077961    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm podName:8ca2cbaa-a242-425a-8886-c327cee392b2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:17.577946492 +0000 UTC m=+39.854715326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qz4bv" (UID: "8ca2cbaa-a242-425a-8886-c327cee392b2") : configmap "kube-rbac-proxy" not found
Sep 15 18:51:17.078021 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:17.077896    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:51:17.078114 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:17.078037    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls podName:8ca2cbaa-a242-425a-8886-c327cee392b2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:17.578024458 +0000 UTC m=+39.854793314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qz4bv" (UID: "8ca2cbaa-a242-425a-8886-c327cee392b2") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:51:17.078172 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.078142    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ca2cbaa-a242-425a-8886-c327cee392b2-data-volume\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:17.089562 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.089529    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czsv7\" (UniqueName: \"kubernetes.io/projected/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-api-access-czsv7\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:17.297951 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.297847    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-q87xv"]
Sep 15 18:51:17.315355 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.315326    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-q87xv"]
Sep 15 18:51:17.315520 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.315434    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q87xv"
Sep 15 18:51:17.317475 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.317444    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 15 18:51:17.317596 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.317478    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 15 18:51:17.317596 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.317488    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cw5rd\""
Sep 15 18:51:17.481416 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.481379    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4jzk\" (UniqueName: \"kubernetes.io/projected/87191718-cfa9-4ccd-98c9-d2dfd1470523-kube-api-access-v4jzk\") pod \"migrator-8b4d697c5-q87xv\" (UID: \"87191718-cfa9-4ccd-98c9-d2dfd1470523\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q87xv"
Sep 15 18:51:17.582532 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.582438    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:17.582532 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.582485    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:17.582532 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.582513    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4jzk\" (UniqueName: \"kubernetes.io/projected/87191718-cfa9-4ccd-98c9-d2dfd1470523-kube-api-access-v4jzk\") pod \"migrator-8b4d697c5-q87xv\" (UID: \"87191718-cfa9-4ccd-98c9-d2dfd1470523\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q87xv"
Sep 15 18:51:17.582816 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:17.582601    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:51:17.582816 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:17.582607    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:51:17.582816 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:17.582674    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm podName:8ca2cbaa-a242-425a-8886-c327cee392b2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:18.582655024 +0000 UTC m=+40.859423876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qz4bv" (UID: "8ca2cbaa-a242-425a-8886-c327cee392b2") : configmap "kube-rbac-proxy" not found
Sep 15 18:51:17.582816 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:17.582722    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls podName:8ca2cbaa-a242-425a-8886-c327cee392b2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:18.582708068 +0000 UTC m=+40.859476919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qz4bv" (UID: "8ca2cbaa-a242-425a-8886-c327cee392b2") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:51:17.589754 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.589727    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4jzk\" (UniqueName: \"kubernetes.io/projected/87191718-cfa9-4ccd-98c9-d2dfd1470523-kube-api-access-v4jzk\") pod \"migrator-8b4d697c5-q87xv\" (UID: \"87191718-cfa9-4ccd-98c9-d2dfd1470523\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q87xv"
Sep 15 18:51:17.624779 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.624750    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q87xv"
Sep 15 18:51:17.907167 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:17.907097    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-q87xv"]
Sep 15 18:51:17.910513 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:51:17.910481    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87191718_cfa9_4ccd_98c9_d2dfd1470523.slice/crio-a58a5568da4a09b16745cd862bc0d4a4df18f4dac06dafc34bada5e73c51d83f WatchSource:0}: Error finding container a58a5568da4a09b16745cd862bc0d4a4df18f4dac06dafc34bada5e73c51d83f: Status 404 returned error can't find the container with id a58a5568da4a09b16745cd862bc0d4a4df18f4dac06dafc34bada5e73c51d83f
Sep 15 18:51:18.212534 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:18.212427    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5n5wz_9fd5a36a-c351-4e64-8b75-abf7e06279a0/dns-node-resolver/0.log"
Sep 15 18:51:18.289631 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:18.289590    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:18.289813 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:18.289737    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:18.289813 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:18.289804    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls podName:69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:26.289788693 +0000 UTC m=+48.566557546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls") pod "dns-default-bq2c6" (UID: "69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:18.390915 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:18.390879    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:18.391056 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:18.391001    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:18.391056 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:18.391012    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dc58f8f87-zjl2r: secret "image-registry-tls" not found
Sep 15 18:51:18.391134 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:18.391061    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls podName:dada676b-2a65-4f7c-b43f-61969e29ad85 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:26.39104741 +0000 UTC m=+48.667816244 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls") pod "image-registry-7dc58f8f87-zjl2r" (UID: "dada676b-2a65-4f7c-b43f-61969e29ad85") : secret "image-registry-tls" not found
Sep 15 18:51:18.541285 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:18.541197    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q87xv" event={"ID":"87191718-cfa9-4ccd-98c9-d2dfd1470523","Type":"ContainerStarted","Data":"a58a5568da4a09b16745cd862bc0d4a4df18f4dac06dafc34bada5e73c51d83f"}
Sep 15 18:51:18.593236 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:18.593200    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:18.593416 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:18.593260    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:18.593416 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:18.593389    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:51:18.593540 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:18.593465    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm podName:8ca2cbaa-a242-425a-8886-c327cee392b2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:20.593445549 +0000 UTC m=+42.870214403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qz4bv" (UID: "8ca2cbaa-a242-425a-8886-c327cee392b2") : configmap "kube-rbac-proxy" not found
Sep 15 18:51:18.593540 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:18.593394    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:51:18.593649 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:18.593561    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls podName:8ca2cbaa-a242-425a-8886-c327cee392b2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:20.593538836 +0000 UTC m=+42.870307686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qz4bv" (UID: "8ca2cbaa-a242-425a-8886-c327cee392b2") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:51:19.213125 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.213095    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-776nl_45a116a2-6657-4785-8769-056a5708db32/node-ca/0.log"
Sep 15 18:51:19.544886 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.544849    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q87xv" event={"ID":"87191718-cfa9-4ccd-98c9-d2dfd1470523","Type":"ContainerStarted","Data":"0df1c98b9b33b5d5d7302747aa20779491f8922e25214b167e370a5ced8b1523"}
Sep 15 18:51:19.559593 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.559566    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-p6sfw"]
Sep 15 18:51:19.582500 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.582475    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-p6sfw"]
Sep 15 18:51:19.582615 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.582574    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-p6sfw"
Sep 15 18:51:19.584924 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.584900    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 15 18:51:19.585015 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.584912    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 15 18:51:19.585732 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.585710    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-x6t59\""
Sep 15 18:51:19.585848 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.585719    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 15 18:51:19.585848 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.585777    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 15 18:51:19.702125 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.702088    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/24b00f8a-2d88-4836-88ef-bd1a16c9b822-signing-key\") pod \"service-ca-568869fdc5-p6sfw\" (UID: \"24b00f8a-2d88-4836-88ef-bd1a16c9b822\") " pod="openshift-service-ca/service-ca-568869fdc5-p6sfw"
Sep 15 18:51:19.702361 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.702137    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5rm8\" (UniqueName: \"kubernetes.io/projected/24b00f8a-2d88-4836-88ef-bd1a16c9b822-kube-api-access-r5rm8\") pod \"service-ca-568869fdc5-p6sfw\" (UID: \"24b00f8a-2d88-4836-88ef-bd1a16c9b822\") " pod="openshift-service-ca/service-ca-568869fdc5-p6sfw"
Sep 15 18:51:19.702361 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.702261    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/24b00f8a-2d88-4836-88ef-bd1a16c9b822-signing-cabundle\") pod \"service-ca-568869fdc5-p6sfw\" (UID: \"24b00f8a-2d88-4836-88ef-bd1a16c9b822\") " pod="openshift-service-ca/service-ca-568869fdc5-p6sfw"
Sep 15 18:51:19.802607 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.802579    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/24b00f8a-2d88-4836-88ef-bd1a16c9b822-signing-key\") pod \"service-ca-568869fdc5-p6sfw\" (UID: \"24b00f8a-2d88-4836-88ef-bd1a16c9b822\") " pod="openshift-service-ca/service-ca-568869fdc5-p6sfw"
Sep 15 18:51:19.802795 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.802626    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5rm8\" (UniqueName: \"kubernetes.io/projected/24b00f8a-2d88-4836-88ef-bd1a16c9b822-kube-api-access-r5rm8\") pod \"service-ca-568869fdc5-p6sfw\" (UID: \"24b00f8a-2d88-4836-88ef-bd1a16c9b822\") " pod="openshift-service-ca/service-ca-568869fdc5-p6sfw"
Sep 15 18:51:19.802867 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.802790    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/24b00f8a-2d88-4836-88ef-bd1a16c9b822-signing-cabundle\") pod \"service-ca-568869fdc5-p6sfw\" (UID: \"24b00f8a-2d88-4836-88ef-bd1a16c9b822\") " pod="openshift-service-ca/service-ca-568869fdc5-p6sfw"
Sep 15 18:51:19.803478 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.803452    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/24b00f8a-2d88-4836-88ef-bd1a16c9b822-signing-cabundle\") pod \"service-ca-568869fdc5-p6sfw\" (UID: \"24b00f8a-2d88-4836-88ef-bd1a16c9b822\") " pod="openshift-service-ca/service-ca-568869fdc5-p6sfw"
Sep 15 18:51:19.805214 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.805191    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/24b00f8a-2d88-4836-88ef-bd1a16c9b822-signing-key\") pod \"service-ca-568869fdc5-p6sfw\" (UID: \"24b00f8a-2d88-4836-88ef-bd1a16c9b822\") " pod="openshift-service-ca/service-ca-568869fdc5-p6sfw"
Sep 15 18:51:19.810133 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.810107    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5rm8\" (UniqueName: \"kubernetes.io/projected/24b00f8a-2d88-4836-88ef-bd1a16c9b822-kube-api-access-r5rm8\") pod \"service-ca-568869fdc5-p6sfw\" (UID: \"24b00f8a-2d88-4836-88ef-bd1a16c9b822\") " pod="openshift-service-ca/service-ca-568869fdc5-p6sfw"
Sep 15 18:51:19.891725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:19.891684    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-p6sfw"
Sep 15 18:51:20.178521 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:20.178453    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-p6sfw"]
Sep 15 18:51:20.181714 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:51:20.181692    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b00f8a_2d88_4836_88ef_bd1a16c9b822.slice/crio-d8eb2b2872219a7022fe015ffd6fe6ba45345cd26f067ae4efd9287e11b7b45f WatchSource:0}: Error finding container d8eb2b2872219a7022fe015ffd6fe6ba45345cd26f067ae4efd9287e11b7b45f: Status 404 returned error can't find the container with id d8eb2b2872219a7022fe015ffd6fe6ba45345cd26f067ae4efd9287e11b7b45f
Sep 15 18:51:20.213500 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:20.213472    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lvxks_8216bdee-a7c9-4523-8732-ba0dbba092fd/serve-healthcheck-canary/0.log"
Sep 15 18:51:20.548577 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:20.548533    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q87xv" event={"ID":"87191718-cfa9-4ccd-98c9-d2dfd1470523","Type":"ContainerStarted","Data":"e698b406f065426fbc41fe3d92f44b91d8c6af9634adea69a5a7a856a31c61b9"}
Sep 15 18:51:20.549694 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:20.549675    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-p6sfw" event={"ID":"24b00f8a-2d88-4836-88ef-bd1a16c9b822","Type":"ContainerStarted","Data":"d8eb2b2872219a7022fe015ffd6fe6ba45345cd26f067ae4efd9287e11b7b45f"}
Sep 15 18:51:20.566531 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:20.566477    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q87xv" podStartSLOduration=2.121706237 podStartE2EDuration="3.566461182s" totalImagesPullingTime="1.444754945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:20.565873924 +0000 UTC m=+42.842642782" watchObservedRunningTime="2026-09-15 18:51:20.566461182 +0000 UTC m=+42.843230038"
Sep 15 18:51:20.609469 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:20.609436    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:20.609594 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:20.609552    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:20.609594 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:20.609556    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:51:20.609668 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:20.609614    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm podName:8ca2cbaa-a242-425a-8886-c327cee392b2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:24.609600907 +0000 UTC m=+46.886369756 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qz4bv" (UID: "8ca2cbaa-a242-425a-8886-c327cee392b2") : configmap "kube-rbac-proxy" not found
Sep 15 18:51:20.609668 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:20.609652    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:51:20.609747 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:20.609696    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls podName:8ca2cbaa-a242-425a-8886-c327cee392b2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:24.609682783 +0000 UTC m=+46.886451617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qz4bv" (UID: "8ca2cbaa-a242-425a-8886-c327cee392b2") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:51:23.559399 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:23.559362    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-p6sfw" event={"ID":"24b00f8a-2d88-4836-88ef-bd1a16c9b822","Type":"ContainerStarted","Data":"66bf1eac63618786ca648711b37bd6cc67c9ad1941097c8a5319f0c4527d7894"}
Sep 15 18:51:23.575305 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:23.575235    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-p6sfw" podStartSLOduration=2.054193555 podStartE2EDuration="4.575215809s" totalImagesPullingTime="2.521022254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:23.57482421 +0000 UTC m=+45.851593066" watchObservedRunningTime="2026-09-15 18:51:23.575215809 +0000 UTC m=+45.851984668"
Sep 15 18:51:24.652770 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:24.652731    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:24.653286 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:24.652799    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:24.653286 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:24.652885    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:51:24.653286 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:24.652913    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:51:24.653286 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:24.652964    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm podName:8ca2cbaa-a242-425a-8886-c327cee392b2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:32.652941254 +0000 UTC m=+54.929710093 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qz4bv" (UID: "8ca2cbaa-a242-425a-8886-c327cee392b2") : configmap "kube-rbac-proxy" not found
Sep 15 18:51:24.653286 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:24.652984    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls podName:8ca2cbaa-a242-425a-8886-c327cee392b2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:32.652973743 +0000 UTC m=+54.929742578 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qz4bv" (UID: "8ca2cbaa-a242-425a-8886-c327cee392b2") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:51:26.365924 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:26.365888    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:26.366360 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:26.366032    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:26.366360 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:26.366097    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls podName:69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:42.366082553 +0000 UTC m=+64.642851423 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls") pod "dns-default-bq2c6" (UID: "69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:26.466929 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:26.466888    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls\") pod \"image-registry-7dc58f8f87-zjl2r\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") " pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:26.467055 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:26.467037    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:26.467091 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:26.467058    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dc58f8f87-zjl2r: secret "image-registry-tls" not found
Sep 15 18:51:26.467131 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:26.467114    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls podName:dada676b-2a65-4f7c-b43f-61969e29ad85 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:42.467099764 +0000 UTC m=+64.743868616 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls") pod "image-registry-7dc58f8f87-zjl2r" (UID: "dada676b-2a65-4f7c-b43f-61969e29ad85") : secret "image-registry-tls" not found
Sep 15 18:51:32.709194 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:32.709137    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:32.709927 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:32.709222    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:32.709927 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:32.709300    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:51:32.709927 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:32.709408    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls podName:8ca2cbaa-a242-425a-8886-c327cee392b2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:48.709390801 +0000 UTC m=+70.986159635 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qz4bv" (UID: "8ca2cbaa-a242-425a-8886-c327cee392b2") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:51:32.709927 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:32.709868    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ca2cbaa-a242-425a-8886-c327cee392b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:37.511616 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:37.511587    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmzls"
Sep 15 18:51:39.434438 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.434400    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dc58f8f87-zjl2r"]
Sep 15 18:51:39.434824 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:39.434605    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r" podUID="dada676b-2a65-4f7c-b43f-61969e29ad85"
Sep 15 18:51:39.454009 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.453978    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"]
Sep 15 18:51:39.456649 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.456632    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b5fc565d6-7wfxm"]
Sep 15 18:51:39.456769 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.456751    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"
Sep 15 18:51:39.458464 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.458446    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"]
Sep 15 18:51:39.458591 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.458570    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b5fc565d6-7wfxm"
Sep 15 18:51:39.459514 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.459485    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 15 18:51:39.459514 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.459485    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 15 18:51:39.459641 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.459490    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 15 18:51:39.459641 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.459522    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 15 18:51:39.460525 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.460512    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.461015 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.460998    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 15 18:51:39.461429 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.461412    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xrbjd\""
Sep 15 18:51:39.462979 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.462964    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 15 18:51:39.463268 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.463246    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 15 18:51:39.463383 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.463287    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 15 18:51:39.463383 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.463376    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 15 18:51:39.467623 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.467605    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b5fc565d6-7wfxm"]
Sep 15 18:51:39.472570 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.472211    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"]
Sep 15 18:51:39.473975 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.473953    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"]
Sep 15 18:51:39.554248 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.554214    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55b68bbff6-rd86t"]
Sep 15 18:51:39.556268 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.556252    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.560478 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.560456    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36aaa584-1fb3-44fe-9946-f9d45f20becf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.560564 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.560488    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/36aaa584-1fb3-44fe-9946-f9d45f20becf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.560564 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.560521    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/36aaa584-1fb3-44fe-9946-f9d45f20becf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.560636 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.560578    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c-klusterlet-config\") pod \"klusterlet-addon-workmgr-74d4457877-89shj\" (UID: \"6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"
Sep 15 18:51:39.560636 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.560608    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/36aaa584-1fb3-44fe-9946-f9d45f20becf-ca\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.560697 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.560637    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snqjb\" (UniqueName: \"kubernetes.io/projected/8feff278-1c11-4356-9e1b-fb0a21e71de2-kube-api-access-snqjb\") pod \"managed-serviceaccount-addon-agent-b5fc565d6-7wfxm\" (UID: \"8feff278-1c11-4356-9e1b-fb0a21e71de2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b5fc565d6-7wfxm"
Sep 15 18:51:39.560740 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.560716    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/36aaa584-1fb3-44fe-9946-f9d45f20becf-hub\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.560775 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.560758    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c-tmp\") pod \"klusterlet-addon-workmgr-74d4457877-89shj\" (UID: \"6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"
Sep 15 18:51:39.560807 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.560785    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8k9l\" (UniqueName: \"kubernetes.io/projected/6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c-kube-api-access-h8k9l\") pod \"klusterlet-addon-workmgr-74d4457877-89shj\" (UID: \"6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"
Sep 15 18:51:39.560839 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.560812    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr845\" (UniqueName: \"kubernetes.io/projected/36aaa584-1fb3-44fe-9946-f9d45f20becf-kube-api-access-dr845\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.560881 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.560869    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8feff278-1c11-4356-9e1b-fb0a21e71de2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b5fc565d6-7wfxm\" (UID: \"8feff278-1c11-4356-9e1b-fb0a21e71de2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b5fc565d6-7wfxm"
Sep 15 18:51:39.574691 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.574667    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55b68bbff6-rd86t"]
Sep 15 18:51:39.598967 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.598935    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:39.604366 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.604342    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:39.661290 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661254    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-bound-sa-token\") pod \"dada676b-2a65-4f7c-b43f-61969e29ad85\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") "
Sep 15 18:51:39.661492 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661301    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klbk8\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-kube-api-access-klbk8\") pod \"dada676b-2a65-4f7c-b43f-61969e29ad85\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") "
Sep 15 18:51:39.661492 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661349    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dada676b-2a65-4f7c-b43f-61969e29ad85-installation-pull-secrets\") pod \"dada676b-2a65-4f7c-b43f-61969e29ad85\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") "
Sep 15 18:51:39.661492 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661396    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dada676b-2a65-4f7c-b43f-61969e29ad85-image-registry-private-configuration\") pod \"dada676b-2a65-4f7c-b43f-61969e29ad85\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") "
Sep 15 18:51:39.661492 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661430    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dada676b-2a65-4f7c-b43f-61969e29ad85-trusted-ca\") pod \"dada676b-2a65-4f7c-b43f-61969e29ad85\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") "
Sep 15 18:51:39.661492 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661465    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dada676b-2a65-4f7c-b43f-61969e29ad85-ca-trust-extracted\") pod \"dada676b-2a65-4f7c-b43f-61969e29ad85\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") "
Sep 15 18:51:39.661492 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661490    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-certificates\") pod \"dada676b-2a65-4f7c-b43f-61969e29ad85\" (UID: \"dada676b-2a65-4f7c-b43f-61969e29ad85\") "
Sep 15 18:51:39.661803 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661597    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36aaa584-1fb3-44fe-9946-f9d45f20becf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.661803 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661630    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6c35301-640a-40d1-bafe-7fb5acbc911b-ca-trust-extracted\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.661803 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661662    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/36aaa584-1fb3-44fe-9946-f9d45f20becf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.661803 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661687    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6c35301-640a-40d1-bafe-7fb5acbc911b-trusted-ca\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.661803 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661711    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6c35301-640a-40d1-bafe-7fb5acbc911b-registry-tls\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.661803 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661748    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/36aaa584-1fb3-44fe-9946-f9d45f20becf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.661803 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661782    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c-klusterlet-config\") pod \"klusterlet-addon-workmgr-74d4457877-89shj\" (UID: \"6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"
Sep 15 18:51:39.662134 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661817    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/36aaa584-1fb3-44fe-9946-f9d45f20becf-ca\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.662134 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661844    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snqjb\" (UniqueName: \"kubernetes.io/projected/8feff278-1c11-4356-9e1b-fb0a21e71de2-kube-api-access-snqjb\") pod \"managed-serviceaccount-addon-agent-b5fc565d6-7wfxm\" (UID: \"8feff278-1c11-4356-9e1b-fb0a21e71de2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b5fc565d6-7wfxm"
Sep 15 18:51:39.662134 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661887    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/36aaa584-1fb3-44fe-9946-f9d45f20becf-hub\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.662134 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661911    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c-tmp\") pod \"klusterlet-addon-workmgr-74d4457877-89shj\" (UID: \"6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"
Sep 15 18:51:39.662134 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661936    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8k9l\" (UniqueName: \"kubernetes.io/projected/6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c-kube-api-access-h8k9l\") pod \"klusterlet-addon-workmgr-74d4457877-89shj\" (UID: \"6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"
Sep 15 18:51:39.662134 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661930    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dada676b-2a65-4f7c-b43f-61969e29ad85-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "dada676b-2a65-4f7c-b43f-61969e29ad85" (UID: "dada676b-2a65-4f7c-b43f-61969e29ad85"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:51:39.662134 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661965    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr845\" (UniqueName: \"kubernetes.io/projected/36aaa584-1fb3-44fe-9946-f9d45f20becf-kube-api-access-dr845\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.662134 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.661983    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "dada676b-2a65-4f7c-b43f-61969e29ad85" (UID: "dada676b-2a65-4f7c-b43f-61969e29ad85"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:51:39.662134 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.662000    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6c35301-640a-40d1-bafe-7fb5acbc911b-bound-sa-token\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.662134 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.662033    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcb79\" (UniqueName: \"kubernetes.io/projected/a6c35301-640a-40d1-bafe-7fb5acbc911b-kube-api-access-lcb79\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.662134 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.662073    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8feff278-1c11-4356-9e1b-fb0a21e71de2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b5fc565d6-7wfxm\" (UID: \"8feff278-1c11-4356-9e1b-fb0a21e71de2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b5fc565d6-7wfxm"
Sep 15 18:51:39.662134 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.662096    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6c35301-640a-40d1-bafe-7fb5acbc911b-installation-pull-secrets\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.662134 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.662127    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6c35301-640a-40d1-bafe-7fb5acbc911b-image-registry-private-configuration\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.662767 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.662191    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6c35301-640a-40d1-bafe-7fb5acbc911b-registry-certificates\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.662767 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.662206    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dada676b-2a65-4f7c-b43f-61969e29ad85-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "dada676b-2a65-4f7c-b43f-61969e29ad85" (UID: "dada676b-2a65-4f7c-b43f-61969e29ad85"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:51:39.662767 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.662659    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c-tmp\") pod \"klusterlet-addon-workmgr-74d4457877-89shj\" (UID: \"6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"
Sep 15 18:51:39.663335 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.662935    2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dada676b-2a65-4f7c-b43f-61969e29ad85-trusted-ca\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:51:39.663335 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.662963    2550 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-certificates\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:51:39.663335 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.663262    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/36aaa584-1fb3-44fe-9946-f9d45f20becf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.664799 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.664761    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dada676b-2a65-4f7c-b43f-61969e29ad85-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "dada676b-2a65-4f7c-b43f-61969e29ad85" (UID: "dada676b-2a65-4f7c-b43f-61969e29ad85"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:51:39.664954 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.664928    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-kube-api-access-klbk8" (OuterVolumeSpecName: "kube-api-access-klbk8") pod "dada676b-2a65-4f7c-b43f-61969e29ad85" (UID: "dada676b-2a65-4f7c-b43f-61969e29ad85"). InnerVolumeSpecName "kube-api-access-klbk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:51:39.665045 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.665027    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "dada676b-2a65-4f7c-b43f-61969e29ad85" (UID: "dada676b-2a65-4f7c-b43f-61969e29ad85"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:51:39.665282 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.665248    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dada676b-2a65-4f7c-b43f-61969e29ad85-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "dada676b-2a65-4f7c-b43f-61969e29ad85" (UID: "dada676b-2a65-4f7c-b43f-61969e29ad85"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:51:39.665438 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.665415    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/36aaa584-1fb3-44fe-9946-f9d45f20becf-hub\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.665719 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.665696    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/36aaa584-1fb3-44fe-9946-f9d45f20becf-ca\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.666214 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.666181    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36aaa584-1fb3-44fe-9946-f9d45f20becf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.666347 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.666308    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c-klusterlet-config\") pod \"klusterlet-addon-workmgr-74d4457877-89shj\" (UID: \"6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"
Sep 15 18:51:39.666605 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.666588    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8feff278-1c11-4356-9e1b-fb0a21e71de2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b5fc565d6-7wfxm\" (UID: \"8feff278-1c11-4356-9e1b-fb0a21e71de2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b5fc565d6-7wfxm"
Sep 15 18:51:39.666709 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.666693    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/36aaa584-1fb3-44fe-9946-f9d45f20becf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.672529 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.672507    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8k9l\" (UniqueName: \"kubernetes.io/projected/6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c-kube-api-access-h8k9l\") pod \"klusterlet-addon-workmgr-74d4457877-89shj\" (UID: \"6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"
Sep 15 18:51:39.672705 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.672688    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snqjb\" (UniqueName: \"kubernetes.io/projected/8feff278-1c11-4356-9e1b-fb0a21e71de2-kube-api-access-snqjb\") pod \"managed-serviceaccount-addon-agent-b5fc565d6-7wfxm\" (UID: \"8feff278-1c11-4356-9e1b-fb0a21e71de2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b5fc565d6-7wfxm"
Sep 15 18:51:39.673248 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.673231    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr845\" (UniqueName: \"kubernetes.io/projected/36aaa584-1fb3-44fe-9946-f9d45f20becf-kube-api-access-dr845\") pod \"cluster-proxy-proxy-agent-6b444969bd-wsnjn\" (UID: \"36aaa584-1fb3-44fe-9946-f9d45f20becf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.764221 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.764140    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6c35301-640a-40d1-bafe-7fb5acbc911b-registry-certificates\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.764221 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.764193    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6c35301-640a-40d1-bafe-7fb5acbc911b-ca-trust-extracted\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.764474 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.764269    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6c35301-640a-40d1-bafe-7fb5acbc911b-trusted-ca\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.764474 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.764305    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6c35301-640a-40d1-bafe-7fb5acbc911b-registry-tls\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.764474 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.764405    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6c35301-640a-40d1-bafe-7fb5acbc911b-bound-sa-token\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.764474 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.764433    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcb79\" (UniqueName: \"kubernetes.io/projected/a6c35301-640a-40d1-bafe-7fb5acbc911b-kube-api-access-lcb79\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.764727 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.764474    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6c35301-640a-40d1-bafe-7fb5acbc911b-installation-pull-secrets\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.764727 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.764506    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6c35301-640a-40d1-bafe-7fb5acbc911b-image-registry-private-configuration\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.764727 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.764554    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6c35301-640a-40d1-bafe-7fb5acbc911b-ca-trust-extracted\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.764727 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.764571    2550 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dada676b-2a65-4f7c-b43f-61969e29ad85-image-registry-private-configuration\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:51:39.764727 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.764589    2550 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dada676b-2a65-4f7c-b43f-61969e29ad85-ca-trust-extracted\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:51:39.764727 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.764604    2550 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-bound-sa-token\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:51:39.764727 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.764618    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klbk8\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-kube-api-access-klbk8\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:51:39.764727 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.764633    2550 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dada676b-2a65-4f7c-b43f-61969e29ad85-installation-pull-secrets\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:51:39.765067 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.765053    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6c35301-640a-40d1-bafe-7fb5acbc911b-registry-certificates\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.765266 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.765243    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6c35301-640a-40d1-bafe-7fb5acbc911b-trusted-ca\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.766926 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.766901    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6c35301-640a-40d1-bafe-7fb5acbc911b-installation-pull-secrets\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.767007 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.766961    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6c35301-640a-40d1-bafe-7fb5acbc911b-registry-tls\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.767045 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.767002    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6c35301-640a-40d1-bafe-7fb5acbc911b-image-registry-private-configuration\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.767742 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.767722    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"
Sep 15 18:51:39.774271 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.774251    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6c35301-640a-40d1-bafe-7fb5acbc911b-bound-sa-token\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.774548 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.774533    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcb79\" (UniqueName: \"kubernetes.io/projected/a6c35301-640a-40d1-bafe-7fb5acbc911b-kube-api-access-lcb79\") pod \"image-registry-55b68bbff6-rd86t\" (UID: \"a6c35301-640a-40d1-bafe-7fb5acbc911b\") " pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:39.783323 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.783294    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b5fc565d6-7wfxm"
Sep 15 18:51:39.789392 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.789366    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"
Sep 15 18:51:39.877618 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.877594    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nk7cf\""
Sep 15 18:51:39.886351 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:39.886328    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:40.038580 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:40.038549    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"]
Sep 15 18:51:40.041435 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:51:40.041408    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7fc7c5_e9d8_41d7_90d9_f231a5a6061c.slice/crio-ec8aaf614aaf3063f449e3f42ac0ee6be723dc452497fab3108b2d9430567a82 WatchSource:0}: Error finding container ec8aaf614aaf3063f449e3f42ac0ee6be723dc452497fab3108b2d9430567a82: Status 404 returned error can't find the container with id ec8aaf614aaf3063f449e3f42ac0ee6be723dc452497fab3108b2d9430567a82
Sep 15 18:51:40.060562 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:40.060535    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b5fc565d6-7wfxm"]
Sep 15 18:51:40.063295 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:51:40.063270    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8feff278_1c11_4356_9e1b_fb0a21e71de2.slice/crio-5d311789bd5eb71d90b4c17cf8c86ea4766f2d19b0c6b6974e31d2e7df1673c1 WatchSource:0}: Error finding container 5d311789bd5eb71d90b4c17cf8c86ea4766f2d19b0c6b6974e31d2e7df1673c1: Status 404 returned error can't find the container with id 5d311789bd5eb71d90b4c17cf8c86ea4766f2d19b0c6b6974e31d2e7df1673c1
Sep 15 18:51:40.068397 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:40.068339    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn"]
Sep 15 18:51:40.070588 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:51:40.070567    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36aaa584_1fb3_44fe_9946_f9d45f20becf.slice/crio-14b69d7080bf1b51e1999c73d20a0f1303f519b85eae03da53a1641b66786ba8 WatchSource:0}: Error finding container 14b69d7080bf1b51e1999c73d20a0f1303f519b85eae03da53a1641b66786ba8: Status 404 returned error can't find the container with id 14b69d7080bf1b51e1999c73d20a0f1303f519b85eae03da53a1641b66786ba8
Sep 15 18:51:40.153624 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:40.153591    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55b68bbff6-rd86t"]
Sep 15 18:51:40.156329 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:51:40.156291    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c35301_640a_40d1_bafe_7fb5acbc911b.slice/crio-2a575a6b200694399e3b8a6453c01251cd45d3751aa0afe0906af7c4938cbe10 WatchSource:0}: Error finding container 2a575a6b200694399e3b8a6453c01251cd45d3751aa0afe0906af7c4938cbe10: Status 404 returned error can't find the container with id 2a575a6b200694399e3b8a6453c01251cd45d3751aa0afe0906af7c4938cbe10
Sep 15 18:51:40.602789 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:40.602696    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55b68bbff6-rd86t" event={"ID":"a6c35301-640a-40d1-bafe-7fb5acbc911b","Type":"ContainerStarted","Data":"a6f66da14d009a035fd720b78ec04d1b368c25f7a1573f392b52c677b5b39dec"}
Sep 15 18:51:40.602789 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:40.602751    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55b68bbff6-rd86t" event={"ID":"a6c35301-640a-40d1-bafe-7fb5acbc911b","Type":"ContainerStarted","Data":"2a575a6b200694399e3b8a6453c01251cd45d3751aa0afe0906af7c4938cbe10"}
Sep 15 18:51:40.603237 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:40.602805    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:51:40.603993 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:40.603968    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn" event={"ID":"36aaa584-1fb3-44fe-9946-f9d45f20becf","Type":"ContainerStarted","Data":"14b69d7080bf1b51e1999c73d20a0f1303f519b85eae03da53a1641b66786ba8"}
Sep 15 18:51:40.605070 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:40.605023    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b5fc565d6-7wfxm" event={"ID":"8feff278-1c11-4356-9e1b-fb0a21e71de2","Type":"ContainerStarted","Data":"5d311789bd5eb71d90b4c17cf8c86ea4766f2d19b0c6b6974e31d2e7df1673c1"}
Sep 15 18:51:40.606041 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:40.606017    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj" event={"ID":"6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c","Type":"ContainerStarted","Data":"ec8aaf614aaf3063f449e3f42ac0ee6be723dc452497fab3108b2d9430567a82"}
Sep 15 18:51:40.606145 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:40.606040    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dc58f8f87-zjl2r"
Sep 15 18:51:40.620328 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:40.620266    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55b68bbff6-rd86t" podStartSLOduration=1.6202513920000001 podStartE2EDuration="1.620251392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:40.619337789 +0000 UTC m=+62.896106644" watchObservedRunningTime="2026-09-15 18:51:40.620251392 +0000 UTC m=+62.897020242"
Sep 15 18:51:40.645299 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:40.643373    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dc58f8f87-zjl2r"]
Sep 15 18:51:40.647178 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:40.647153    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7dc58f8f87-zjl2r"]
Sep 15 18:51:40.672833 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:40.672779    2550 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dada676b-2a65-4f7c-b43f-61969e29ad85-registry-tls\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:51:42.295052 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:42.294889    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dada676b-2a65-4f7c-b43f-61969e29ad85" path="/var/lib/kubelet/pods/dada676b-2a65-4f7c-b43f-61969e29ad85/volumes"
Sep 15 18:51:42.390692 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:42.390246    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:42.393647 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:42.393584    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31-metrics-tls\") pod \"dns-default-bq2c6\" (UID: \"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31\") " pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:42.672520 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:42.672443    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcddk\""
Sep 15 18:51:42.681295 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:42.681265    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:42.895140 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:42.895082    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs\") pod \"network-metrics-daemon-9tr5r\" (UID: \"66062b2c-ba98-4b6a-bc2c-656a6c7c339b\") " pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:51:42.895338 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:42.895165    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjsrq\" (UniqueName: \"kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq\") pod \"network-check-target-8jngj\" (UID: \"540b4731-6750-43d4-96b3-759aec9a4eee\") " pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:51:42.897779 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:42.897752    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 18:51:42.898363 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:42.898341    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 18:51:42.908704 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:42.908676    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 18:51:42.909786 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:42.909762    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66062b2c-ba98-4b6a-bc2c-656a6c7c339b-metrics-certs\") pod \"network-metrics-daemon-9tr5r\" (UID: \"66062b2c-ba98-4b6a-bc2c-656a6c7c339b\") " pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:51:42.922158 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:42.922095    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjsrq\" (UniqueName: \"kubernetes.io/projected/540b4731-6750-43d4-96b3-759aec9a4eee-kube-api-access-kjsrq\") pod \"network-check-target-8jngj\" (UID: \"540b4731-6750-43d4-96b3-759aec9a4eee\") " pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:51:42.995897 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:42.995798    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:51:42.998301 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:42.998076    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 18:51:43.009610 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:43.009553    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae-original-pull-secret\") pod \"global-pull-secret-syncer-vgmvv\" (UID: \"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae\") " pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:51:43.104432 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:43.104404    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gnvqj\""
Sep 15 18:51:43.110794 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:43.110757    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4h8bk\""
Sep 15 18:51:43.112451 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:43.112407    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tr5r"
Sep 15 18:51:43.112589 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:43.112484    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vgmvv"
Sep 15 18:51:43.118559 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:43.118538    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:51:46.875016 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:46.874987    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bq2c6"]
Sep 15 18:51:46.877615 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:51:46.877583    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a08e4e_d5c2_4e7d_bfd8_61e8f8325b31.slice/crio-ec5b6d6342744bec2936a902359e580fabdfa75395abdc2dc1fbc380dfcd49f9 WatchSource:0}: Error finding container ec5b6d6342744bec2936a902359e580fabdfa75395abdc2dc1fbc380dfcd49f9: Status 404 returned error can't find the container with id ec5b6d6342744bec2936a902359e580fabdfa75395abdc2dc1fbc380dfcd49f9
Sep 15 18:51:46.889912 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:46.889890    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9tr5r"]
Sep 15 18:51:46.892551 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:51:46.892527    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66062b2c_ba98_4b6a_bc2c_656a6c7c339b.slice/crio-b669e1abb834560a7c83f7c1b7e21353aa9f84473bc7170f7f320882240d0002 WatchSource:0}: Error finding container b669e1abb834560a7c83f7c1b7e21353aa9f84473bc7170f7f320882240d0002: Status 404 returned error can't find the container with id b669e1abb834560a7c83f7c1b7e21353aa9f84473bc7170f7f320882240d0002
Sep 15 18:51:46.898567 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:46.898546    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vgmvv"]
Sep 15 18:51:46.901134 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:51:46.901116    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05cb8d3_8b0d_45c8_aeb2_a7ecb44a08ae.slice/crio-687c29497b01807642df37b5bb5512833f002e6fc5ed11d7315e89962ec12e14 WatchSource:0}: Error finding container 687c29497b01807642df37b5bb5512833f002e6fc5ed11d7315e89962ec12e14: Status 404 returned error can't find the container with id 687c29497b01807642df37b5bb5512833f002e6fc5ed11d7315e89962ec12e14
Sep 15 18:51:46.931452 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:46.931424    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8jngj"]
Sep 15 18:51:46.933613 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:51:46.933594    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540b4731_6750_43d4_96b3_759aec9a4eee.slice/crio-b508c388f8df93f72f28ab7b3b3f9c2467555f3b77e29c41bdc0091694e72a12 WatchSource:0}: Error finding container b508c388f8df93f72f28ab7b3b3f9c2467555f3b77e29c41bdc0091694e72a12: Status 404 returned error can't find the container with id b508c388f8df93f72f28ab7b3b3f9c2467555f3b77e29c41bdc0091694e72a12
Sep 15 18:51:47.632169 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:47.632123    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b5fc565d6-7wfxm" event={"ID":"8feff278-1c11-4356-9e1b-fb0a21e71de2","Type":"ContainerStarted","Data":"2ce2f7aa78ca5819c1c56aaf92ec17568d44d122328e00105fd5b87b221bb2de"}
Sep 15 18:51:47.635508 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:47.635482    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj" event={"ID":"6d7fc7c5-e9d8-41d7-90d9-f231a5a6061c","Type":"ContainerStarted","Data":"dcb08d33efd4bdb93826027334e190934f1b47e04222c4879316ad76445f2ac4"}
Sep 15 18:51:47.636185 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:47.636155    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"
Sep 15 18:51:47.637782 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:47.637742    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj"
Sep 15 18:51:47.639461 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:47.639413    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vgmvv" event={"ID":"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae","Type":"ContainerStarted","Data":"687c29497b01807642df37b5bb5512833f002e6fc5ed11d7315e89962ec12e14"}
Sep 15 18:51:47.643255 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:47.642729    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn" event={"ID":"36aaa584-1fb3-44fe-9946-f9d45f20becf","Type":"ContainerStarted","Data":"060535e8948c5d6fd490367c8004a9467c7a9617aff4b5ec97a40103b924c2b0"}
Sep 15 18:51:47.643255 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:47.642763    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn" event={"ID":"36aaa584-1fb3-44fe-9946-f9d45f20becf","Type":"ContainerStarted","Data":"7abec2e837edc9b38ed6a29f697e67a05a04040a209ac28867a04d4b295ac2cb"}
Sep 15 18:51:47.643255 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:47.642780    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn" event={"ID":"36aaa584-1fb3-44fe-9946-f9d45f20becf","Type":"ContainerStarted","Data":"52bb4e8d7b24cd047ea2c57c192ce04230cd156b63369f90ea6b6ed74efbdbdb"}
Sep 15 18:51:47.644830 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:47.644808    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8jngj" event={"ID":"540b4731-6750-43d4-96b3-759aec9a4eee","Type":"ContainerStarted","Data":"b508c388f8df93f72f28ab7b3b3f9c2467555f3b77e29c41bdc0091694e72a12"}
Sep 15 18:51:47.646384 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:47.646360    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9tr5r" event={"ID":"66062b2c-ba98-4b6a-bc2c-656a6c7c339b","Type":"ContainerStarted","Data":"b669e1abb834560a7c83f7c1b7e21353aa9f84473bc7170f7f320882240d0002"}
Sep 15 18:51:47.650198 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:47.647917    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bq2c6" event={"ID":"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31","Type":"ContainerStarted","Data":"ec5b6d6342744bec2936a902359e580fabdfa75395abdc2dc1fbc380dfcd49f9"}
Sep 15 18:51:47.668133 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:47.668083    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b444969bd-wsnjn" podStartSLOduration=2.130336341 podStartE2EDuration="8.668064891s" totalImagesPullingTime="6.53772855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:47.66707916 +0000 UTC m=+69.943848018" watchObservedRunningTime="2026-09-15 18:51:47.668064891 +0000 UTC m=+69.944833747"
Sep 15 18:51:47.668768 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:47.668732    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b5fc565d6-7wfxm" podStartSLOduration=2.1289603440000002 podStartE2EDuration="8.668724458s" totalImagesPullingTime="6.539764114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:47.648051811 +0000 UTC m=+69.924820668" watchObservedRunningTime="2026-09-15 18:51:47.668724458 +0000 UTC m=+69.945493315"
Sep 15 18:51:47.682343 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:47.682033    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d4457877-89shj" podStartSLOduration=2.105555282 podStartE2EDuration="8.682017287s" totalImagesPullingTime="6.576462005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:47.680738644 +0000 UTC m=+69.957507505" watchObservedRunningTime="2026-09-15 18:51:47.682017287 +0000 UTC m=+69.958786135"
Sep 15 18:51:48.749153 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:48.748365    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:48.752132 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:48.752086    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ca2cbaa-a242-425a-8886-c327cee392b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qz4bv\" (UID: \"8ca2cbaa-a242-425a-8886-c327cee392b2\") " pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:48.989853 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:48.989820    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pj5l2\""
Sep 15 18:51:48.998823 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:48.998687    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qz4bv"
Sep 15 18:51:49.395098 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:49.395008    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qz4bv"]
Sep 15 18:51:49.657725 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:49.657689    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9tr5r" event={"ID":"66062b2c-ba98-4b6a-bc2c-656a6c7c339b","Type":"ContainerStarted","Data":"be9751e0f743e706c940c182755ce855b37d948d395f20b5828135259d93e587"}
Sep 15 18:51:49.659709 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:49.659650    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bq2c6" event={"ID":"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31","Type":"ContainerStarted","Data":"b0ac161ec1d75cecae530bca7d4d2b9152a065aded671826938f3bf5be157ee8"}
Sep 15 18:51:50.664498 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:50.664420    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qz4bv" event={"ID":"8ca2cbaa-a242-425a-8886-c327cee392b2","Type":"ContainerStarted","Data":"72ae1b5cb4c4c886f52bcd38b903ba535b35629666508721dff0fe9bbfb88a9e"}
Sep 15 18:51:52.672185 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:52.672145    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vgmvv" event={"ID":"f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae","Type":"ContainerStarted","Data":"ba10f11789986f6473fdc9aa63e060e261584eddd1e3356ea2f8adb8a3378757"}
Sep 15 18:51:52.673619 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:52.673595    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8jngj" event={"ID":"540b4731-6750-43d4-96b3-759aec9a4eee","Type":"ContainerStarted","Data":"2306bfde4eac2cc2a811bd2352add643f458374ee70b9667e7c0b43f8131ad77"}
Sep 15 18:51:52.673742 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:52.673658    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:51:52.675463 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:52.675440    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9tr5r" event={"ID":"66062b2c-ba98-4b6a-bc2c-656a6c7c339b","Type":"ContainerStarted","Data":"5d0d81fa8697af8b7c20d575f2a6f7ce4eece4e73398e4913bfbc5a2779742de"}
Sep 15 18:51:52.677058 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:52.677038    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bq2c6" event={"ID":"69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31","Type":"ContainerStarted","Data":"e2ac04104a1a7570a17890460099e08f5537ef9a8575d83a1060d8b53257b3f9"}
Sep 15 18:51:52.677131 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:52.677103    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:51:52.678475 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:52.678453    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qz4bv" event={"ID":"8ca2cbaa-a242-425a-8886-c327cee392b2","Type":"ContainerStarted","Data":"ab2ee46a8bba46db905a09255ca2af300b6aff25555d305f186a393e7f814a85"}
Sep 15 18:51:52.686092 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:52.686055    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vgmvv" podStartSLOduration=69.866076189 podStartE2EDuration="1m14.686042939s" totalImagesPullingTime="4.81996675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:52.685026084 +0000 UTC m=+74.961794942" watchObservedRunningTime="2026-09-15 18:51:52.686042939 +0000 UTC m=+74.962811795"
Sep 15 18:51:52.700212 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:52.700168    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bq2c6" podStartSLOduration=40.450839924 podStartE2EDuration="42.700156115s" totalImagesPullingTime="2.249316191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:52.699252688 +0000 UTC m=+74.976021546" watchObservedRunningTime="2026-09-15 18:51:52.700156115 +0000 UTC m=+74.976924971"
Sep 15 18:51:52.712343 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:52.712280    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9tr5r" podStartSLOduration=72.483742499 podStartE2EDuration="1m14.712265519s" totalImagesPullingTime="2.22852302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:52.712077303 +0000 UTC m=+74.988846159" watchObservedRunningTime="2026-09-15 18:51:52.712265519 +0000 UTC m=+74.989034377"
Sep 15 18:51:52.728430 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:52.728394    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8jngj" podStartSLOduration=69.948852796 podStartE2EDuration="1m14.728381416s" totalImagesPullingTime="4.77952862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:52.727678679 +0000 UTC m=+75.004447536" watchObservedRunningTime="2026-09-15 18:51:52.728381416 +0000 UTC m=+75.005150271"
Sep 15 18:51:53.685585 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:53.685538    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qz4bv" event={"ID":"8ca2cbaa-a242-425a-8886-c327cee392b2","Type":"ContainerStarted","Data":"f0c97d6fa416b88372140ab5def8e24b36a9a39f52c59babc18f7690922c357d"}
Sep 15 18:51:54.263230 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.263205    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rqcwz"]
Sep 15 18:51:54.266628 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.266607    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.269305 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.269200    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 18:51:54.269429 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.269334    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 18:51:54.269493 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.269469    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 18:51:54.269759 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.269603    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 18:51:54.269759 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.269676    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 18:51:54.270166 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.270059    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n5z5v\""
Sep 15 18:51:54.270166 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.270076    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 18:51:54.391488 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.391445    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-wtmp\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.391653 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.391496    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0dfcf813-4f23-49da-a941-71fecec82235-root\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.391653 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.391545    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dfcf813-4f23-49da-a941-71fecec82235-metrics-client-ca\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.391791 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.391666    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-textfile\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.391791 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.391702    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kvqc\" (UniqueName: \"kubernetes.io/projected/0dfcf813-4f23-49da-a941-71fecec82235-kube-api-access-8kvqc\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.391791 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.391733    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.391791 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.391761    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-accelerators-collector-config\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.391921 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.391829    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-tls\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.391957 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.391919    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dfcf813-4f23-49da-a941-71fecec82235-sys\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.492699 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.492600    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dfcf813-4f23-49da-a941-71fecec82235-sys\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.493053 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.493029    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-wtmp\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.493194 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.493179    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0dfcf813-4f23-49da-a941-71fecec82235-root\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.494449 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.493196    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-wtmp\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.494449 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.492946    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dfcf813-4f23-49da-a941-71fecec82235-sys\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.494449 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.493240    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0dfcf813-4f23-49da-a941-71fecec82235-root\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.494449 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.493329    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dfcf813-4f23-49da-a941-71fecec82235-metrics-client-ca\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.494449 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.493480    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-textfile\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.494449 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.493511    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kvqc\" (UniqueName: \"kubernetes.io/projected/0dfcf813-4f23-49da-a941-71fecec82235-kube-api-access-8kvqc\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.494449 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.493539    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.494449 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.493874    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-textfile\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.494449 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.493923    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-accelerators-collector-config\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.494449 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.493954    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dfcf813-4f23-49da-a941-71fecec82235-metrics-client-ca\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.494449 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.493962    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-tls\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.494449 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:54.494092    2550 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 15 18:51:54.494449 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:51:54.494151    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-tls podName:0dfcf813-4f23-49da-a941-71fecec82235 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:54.994129634 +0000 UTC m=+77.270898471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-tls") pod "node-exporter-rqcwz" (UID: "0dfcf813-4f23-49da-a941-71fecec82235") : secret "node-exporter-tls" not found
Sep 15 18:51:54.494449 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.494411    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-accelerators-collector-config\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.499069 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.498598    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.507912 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.507871    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kvqc\" (UniqueName: \"kubernetes.io/projected/0dfcf813-4f23-49da-a941-71fecec82235-kube-api-access-8kvqc\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:54.691126 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.691085    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qz4bv" event={"ID":"8ca2cbaa-a242-425a-8886-c327cee392b2","Type":"ContainerStarted","Data":"e4a3e442f71e659de33aaa4efae4a7203bd4b55766abc1a93589b557baf2da87"}
Sep 15 18:51:54.709012 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.708962    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qz4bv" podStartSLOduration=36.372743312 podStartE2EDuration="38.708945361s" totalImagesPullingTime="2.336202049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:51:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:54.708278624 +0000 UTC m=+76.985047492" watchObservedRunningTime="2026-09-15 18:51:54.708945361 +0000 UTC m=+76.985714217"
Sep 15 18:51:54.998903 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:54.998869    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-tls\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:55.001607 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:55.001583    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0dfcf813-4f23-49da-a941-71fecec82235-node-exporter-tls\") pod \"node-exporter-rqcwz\" (UID: \"0dfcf813-4f23-49da-a941-71fecec82235\") " pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:55.176553 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:55.176495    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rqcwz"
Sep 15 18:51:55.189304 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:51:55.189272    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dfcf813_4f23_49da_a941_71fecec82235.slice/crio-081264e841b91368abe34a2ae73ebd2e4b3d92dab21923155d867ef6759affc4 WatchSource:0}: Error finding container 081264e841b91368abe34a2ae73ebd2e4b3d92dab21923155d867ef6759affc4: Status 404 returned error can't find the container with id 081264e841b91368abe34a2ae73ebd2e4b3d92dab21923155d867ef6759affc4
Sep 15 18:51:55.696605 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:55.696559    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rqcwz" event={"ID":"0dfcf813-4f23-49da-a941-71fecec82235","Type":"ContainerStarted","Data":"081264e841b91368abe34a2ae73ebd2e4b3d92dab21923155d867ef6759affc4"}
Sep 15 18:51:56.702032 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:56.701938    2550 generic.go:356] "Generic (PLEG): container finished" podID="0dfcf813-4f23-49da-a941-71fecec82235" containerID="fe0a1eca2064a6d9b38e70988370d1f1300ca276b4789d8790f5f031eacd17ab" exitCode=0
Sep 15 18:51:56.702032 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:56.701975    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rqcwz" event={"ID":"0dfcf813-4f23-49da-a941-71fecec82235","Type":"ContainerDied","Data":"fe0a1eca2064a6d9b38e70988370d1f1300ca276b4789d8790f5f031eacd17ab"}
Sep 15 18:51:57.708611 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:57.708571    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rqcwz" event={"ID":"0dfcf813-4f23-49da-a941-71fecec82235","Type":"ContainerStarted","Data":"50d934d44afe6b0a351ebf071006e85e3d24536971bf88a5f84c8320b8bcd0e6"}
Sep 15 18:51:57.708979 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:57.708632    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rqcwz" event={"ID":"0dfcf813-4f23-49da-a941-71fecec82235","Type":"ContainerStarted","Data":"135bf7ec844a01b080b7568e129212c583ec10f9bb148cfc4a6c16568af868b4"}
Sep 15 18:51:57.726569 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:57.726520    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rqcwz" podStartSLOduration=2.564158522 podStartE2EDuration="3.726502619s" totalImagesPullingTime="1.098011298s" totalInitContainerRuntime="64.332799ms" isStatefulPod=true podCreationTimestamp="2026-09-15 18:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:57.725064823 +0000 UTC m=+80.001833680" watchObservedRunningTime="2026-09-15 18:51:57.726502619 +0000 UTC m=+80.003271474"
Sep 15 18:51:59.688758 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:51:59.688721    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bq2c6"
Sep 15 18:52:01.614185 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:52:01.614076    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55b68bbff6-rd86t"
Sep 15 18:52:23.137478 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:52:23.137444    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lvxks_8216bdee-a7c9-4523-8732-ba0dbba092fd/serve-healthcheck-canary/0.log"
Sep 15 18:52:23.688607 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:52:23.688576    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8jngj"
Sep 15 18:55:38.090640 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:55:38.090610    2550 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 18:56:44.765095 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:44.765023    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"]
Sep 15 18:56:44.767092 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:44.767075    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:44.769633 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:44.769609    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 18:56:44.769772 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:44.769677    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 18:56:44.769772 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:44.769744    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h74qm\""
Sep 15 18:56:44.771389 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:44.770463    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 18:56:44.774977 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:44.774954    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 18:56:44.803815 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:44.803793    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"]
Sep 15 18:56:44.915811 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:44.915778    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fc2947b-2cbe-4f61-8b52-7231498ceac3-webhook-cert\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-z5sbq\" (UID: \"1fc2947b-2cbe-4f61-8b52-7231498ceac3\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:44.915965 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:44.915820    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fc2947b-2cbe-4f61-8b52-7231498ceac3-tmp\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-z5sbq\" (UID: \"1fc2947b-2cbe-4f61-8b52-7231498ceac3\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:44.915965 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:44.915924    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1fc2947b-2cbe-4f61-8b52-7231498ceac3-manifests\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-z5sbq\" (UID: \"1fc2947b-2cbe-4f61-8b52-7231498ceac3\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:44.916045 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:44.915964    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvxrp\" (UniqueName: \"kubernetes.io/projected/1fc2947b-2cbe-4f61-8b52-7231498ceac3-kube-api-access-qvxrp\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-z5sbq\" (UID: \"1fc2947b-2cbe-4f61-8b52-7231498ceac3\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:44.916045 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:44.916010    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1fc2947b-2cbe-4f61-8b52-7231498ceac3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-z5sbq\" (UID: \"1fc2947b-2cbe-4f61-8b52-7231498ceac3\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:45.016634 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:45.016553    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fc2947b-2cbe-4f61-8b52-7231498ceac3-webhook-cert\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-z5sbq\" (UID: \"1fc2947b-2cbe-4f61-8b52-7231498ceac3\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:45.016634 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:45.016589    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fc2947b-2cbe-4f61-8b52-7231498ceac3-tmp\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-z5sbq\" (UID: \"1fc2947b-2cbe-4f61-8b52-7231498ceac3\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:45.016634 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:45.016624    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1fc2947b-2cbe-4f61-8b52-7231498ceac3-manifests\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-z5sbq\" (UID: \"1fc2947b-2cbe-4f61-8b52-7231498ceac3\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:45.016905 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:45.016659    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvxrp\" (UniqueName: \"kubernetes.io/projected/1fc2947b-2cbe-4f61-8b52-7231498ceac3-kube-api-access-qvxrp\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-z5sbq\" (UID: \"1fc2947b-2cbe-4f61-8b52-7231498ceac3\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:45.016905 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:45.016708    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1fc2947b-2cbe-4f61-8b52-7231498ceac3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-z5sbq\" (UID: \"1fc2947b-2cbe-4f61-8b52-7231498ceac3\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:45.017077 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:45.017048    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1fc2947b-2cbe-4f61-8b52-7231498ceac3-manifests\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-z5sbq\" (UID: \"1fc2947b-2cbe-4f61-8b52-7231498ceac3\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:45.019030 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:45.019004    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fc2947b-2cbe-4f61-8b52-7231498ceac3-tmp\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-z5sbq\" (UID: \"1fc2947b-2cbe-4f61-8b52-7231498ceac3\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:45.019127 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:45.019107    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fc2947b-2cbe-4f61-8b52-7231498ceac3-webhook-cert\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-z5sbq\" (UID: \"1fc2947b-2cbe-4f61-8b52-7231498ceac3\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:45.019189 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:45.019173    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1fc2947b-2cbe-4f61-8b52-7231498ceac3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-z5sbq\" (UID: \"1fc2947b-2cbe-4f61-8b52-7231498ceac3\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:45.029296 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:45.029271    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvxrp\" (UniqueName: \"kubernetes.io/projected/1fc2947b-2cbe-4f61-8b52-7231498ceac3-kube-api-access-qvxrp\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-z5sbq\" (UID: \"1fc2947b-2cbe-4f61-8b52-7231498ceac3\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:45.081252 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:45.081228    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:45.360569 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:45.360503    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"]
Sep 15 18:56:45.363199 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:56:45.363174    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc2947b_2cbe_4f61_8b52_7231498ceac3.slice/crio-d9fefab9b87e05f8d11c3da71ada31632c1634aa6036ea1d38a8f9d10ee5c269 WatchSource:0}: Error finding container d9fefab9b87e05f8d11c3da71ada31632c1634aa6036ea1d38a8f9d10ee5c269: Status 404 returned error can't find the container with id d9fefab9b87e05f8d11c3da71ada31632c1634aa6036ea1d38a8f9d10ee5c269
Sep 15 18:56:45.364986 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:45.364968    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 18:56:45.650110 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:45.650045    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq" event={"ID":"1fc2947b-2cbe-4f61-8b52-7231498ceac3","Type":"ContainerStarted","Data":"d9fefab9b87e05f8d11c3da71ada31632c1634aa6036ea1d38a8f9d10ee5c269"}
Sep 15 18:56:48.663624 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:48.663534    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq" event={"ID":"1fc2947b-2cbe-4f61-8b52-7231498ceac3","Type":"ContainerStarted","Data":"11c83ed2d58c82c0a00ffcd4fded7062fa5278a49e9b9c2120f3306d0ce9ef05"}
Sep 15 18:56:49.668693 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:49.668662    2550 generic.go:356] "Generic (PLEG): container finished" podID="1fc2947b-2cbe-4f61-8b52-7231498ceac3" containerID="11c83ed2d58c82c0a00ffcd4fded7062fa5278a49e9b9c2120f3306d0ce9ef05" exitCode=0
Sep 15 18:56:49.669085 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:49.668754    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq" event={"ID":"1fc2947b-2cbe-4f61-8b52-7231498ceac3","Type":"ContainerDied","Data":"11c83ed2d58c82c0a00ffcd4fded7062fa5278a49e9b9c2120f3306d0ce9ef05"}
Sep 15 18:56:50.673928 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:50.673891    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq" event={"ID":"1fc2947b-2cbe-4f61-8b52-7231498ceac3","Type":"ContainerStarted","Data":"2d6b0104d22818c54d4add5ba3fa6e903f7a5da31f1da02a6df26abe1da5285a"}
Sep 15 18:56:50.674449 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:50.674005    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:56:50.693570 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:56:50.693517    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq" podStartSLOduration=2.246811226 podStartE2EDuration="6.693498243s" totalImagesPullingTime="3.406366103s" totalInitContainerRuntime="1.040320914s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:56:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:56:50.690887107 +0000 UTC m=+372.967655975" watchObservedRunningTime="2026-09-15 18:56:50.693498243 +0000 UTC m=+372.970267100"
Sep 15 18:57:01.678847 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:01.678818    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-z5sbq"
Sep 15 18:57:04.676452 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.676415    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8"]
Sep 15 18:57:04.681939 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.681913    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8"
Sep 15 18:57:04.685643 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.685617    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 18:57:04.686669 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.686528    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 18:57:04.686669 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.686540    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 18:57:04.686669 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.686568    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 18:57:04.686669 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.686600    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w2hsw\""
Sep 15 18:57:04.703503 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.703478    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8"]
Sep 15 18:57:04.754668 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.754644    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cec8a25b-eef6-4414-afbf-849f642211f4-tls-certs\") pod \"kube-auth-proxy-5b76b67bf4-tsmb8\" (UID: \"cec8a25b-eef6-4414-afbf-849f642211f4\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8"
Sep 15 18:57:04.754796 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.754702    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6kxx\" (UniqueName: \"kubernetes.io/projected/cec8a25b-eef6-4414-afbf-849f642211f4-kube-api-access-m6kxx\") pod \"kube-auth-proxy-5b76b67bf4-tsmb8\" (UID: \"cec8a25b-eef6-4414-afbf-849f642211f4\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8"
Sep 15 18:57:04.754862 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.754790    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cec8a25b-eef6-4414-afbf-849f642211f4-tmp\") pod \"kube-auth-proxy-5b76b67bf4-tsmb8\" (UID: \"cec8a25b-eef6-4414-afbf-849f642211f4\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8"
Sep 15 18:57:04.855644 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.855616    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cec8a25b-eef6-4414-afbf-849f642211f4-tls-certs\") pod \"kube-auth-proxy-5b76b67bf4-tsmb8\" (UID: \"cec8a25b-eef6-4414-afbf-849f642211f4\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8"
Sep 15 18:57:04.855771 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.855656    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6kxx\" (UniqueName: \"kubernetes.io/projected/cec8a25b-eef6-4414-afbf-849f642211f4-kube-api-access-m6kxx\") pod \"kube-auth-proxy-5b76b67bf4-tsmb8\" (UID: \"cec8a25b-eef6-4414-afbf-849f642211f4\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8"
Sep 15 18:57:04.855771 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.855682    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cec8a25b-eef6-4414-afbf-849f642211f4-tmp\") pod \"kube-auth-proxy-5b76b67bf4-tsmb8\" (UID: \"cec8a25b-eef6-4414-afbf-849f642211f4\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8"
Sep 15 18:57:04.857846 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.857815    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cec8a25b-eef6-4414-afbf-849f642211f4-tmp\") pod \"kube-auth-proxy-5b76b67bf4-tsmb8\" (UID: \"cec8a25b-eef6-4414-afbf-849f642211f4\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8"
Sep 15 18:57:04.858085 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.858065    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cec8a25b-eef6-4414-afbf-849f642211f4-tls-certs\") pod \"kube-auth-proxy-5b76b67bf4-tsmb8\" (UID: \"cec8a25b-eef6-4414-afbf-849f642211f4\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8"
Sep 15 18:57:04.865295 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.865270    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6kxx\" (UniqueName: \"kubernetes.io/projected/cec8a25b-eef6-4414-afbf-849f642211f4-kube-api-access-m6kxx\") pod \"kube-auth-proxy-5b76b67bf4-tsmb8\" (UID: \"cec8a25b-eef6-4414-afbf-849f642211f4\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8"
Sep 15 18:57:04.992054 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:04.991980    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8"
Sep 15 18:57:05.262771 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.262703    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8"]
Sep 15 18:57:05.265924 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:57:05.265900    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec8a25b_eef6_4414_afbf_849f642211f4.slice/crio-d39133d2df2644b5c710563fa4b65772e3dd269879b468b7ee4e35264d9a1b0f WatchSource:0}: Error finding container d39133d2df2644b5c710563fa4b65772e3dd269879b468b7ee4e35264d9a1b0f: Status 404 returned error can't find the container with id d39133d2df2644b5c710563fa4b65772e3dd269879b468b7ee4e35264d9a1b0f
Sep 15 18:57:05.729378 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.729343    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8" event={"ID":"cec8a25b-eef6-4414-afbf-849f642211f4","Type":"ContainerStarted","Data":"d39133d2df2644b5c710563fa4b65772e3dd269879b468b7ee4e35264d9a1b0f"}
Sep 15 18:57:05.928787 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.928753    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"]
Sep 15 18:57:05.931560 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.931544    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:05.933683 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.933658    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 18:57:05.933799 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.933710    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 15 18:57:05.933799 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.933729    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 15 18:57:05.933799 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.933760    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 15 18:57:05.933799 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.933736    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nznzx\""
Sep 15 18:57:05.944571 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.944552    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"]
Sep 15 18:57:05.964047 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.964021    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kc4q\" (UniqueName: \"kubernetes.io/projected/62d73c51-cdb6-453c-af2a-6f77869612b7-kube-api-access-4kc4q\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:05.964128 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.964054    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:05.964128 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.964082    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d73c51-cdb6-453c-af2a-6f77869612b7-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:05.964216 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.964129    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:05.964216 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.964145    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:05.964216 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.964186    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:05.964341 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:05.964220    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.064555 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.064520    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.064661 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.064570    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d73c51-cdb6-453c-af2a-6f77869612b7-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.064661 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.064623    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.064661 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.064652    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.064818 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.064695    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.064818 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.064730    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.064911 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.064811    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kc4q\" (UniqueName: \"kubernetes.io/projected/62d73c51-cdb6-453c-af2a-6f77869612b7-kube-api-access-4kc4q\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.065350 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.065326    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.066822 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.066803    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d73c51-cdb6-453c-af2a-6f77869612b7-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.067027 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.067007    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.067066 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.067031    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.067066 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.067032    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.071488 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.071462    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.071702 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.071686    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kc4q\" (UniqueName: \"kubernetes.io/projected/62d73c51-cdb6-453c-af2a-6f77869612b7-kube-api-access-4kc4q\") pod \"istiod-openshift-gateway-7b7995d5cd-868tl\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.240780 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.240747    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:06.517598 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.517518    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"]
Sep 15 18:57:06.521965 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:57:06.521924    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d73c51_cdb6_453c_af2a_6f77869612b7.slice/crio-454944c8c9fbb569d967c5b322670097d10da878dc31de8ec91f4d0b19607120 WatchSource:0}: Error finding container 454944c8c9fbb569d967c5b322670097d10da878dc31de8ec91f4d0b19607120: Status 404 returned error can't find the container with id 454944c8c9fbb569d967c5b322670097d10da878dc31de8ec91f4d0b19607120
Sep 15 18:57:06.733960 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:06.733929    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl" event={"ID":"62d73c51-cdb6-453c-af2a-6f77869612b7","Type":"ContainerStarted","Data":"454944c8c9fbb569d967c5b322670097d10da878dc31de8ec91f4d0b19607120"}
Sep 15 18:57:10.463360 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:10.463299    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 18:57:10.463719 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:10.463404    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 18:57:10.753555 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:10.753450    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8" event={"ID":"cec8a25b-eef6-4414-afbf-849f642211f4","Type":"ContainerStarted","Data":"0734c4cd98b2aed23108adb5e202460485f519a41ebd31600b14d132810cf652"}
Sep 15 18:57:10.755002 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:10.754973    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl" event={"ID":"62d73c51-cdb6-453c-af2a-6f77869612b7","Type":"ContainerStarted","Data":"8f5b71588738a621926bfc097162deb03c31a119151fb0f6b5264490a4a4de79"}
Sep 15 18:57:10.755122 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:10.755086    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:10.769711 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:10.769666    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-tsmb8" podStartSLOduration=1.572025056 podStartE2EDuration="6.769651682s" totalImagesPullingTime="5.197626626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:57:10.767945302 +0000 UTC m=+393.044714157" watchObservedRunningTime="2026-09-15 18:57:10.769651682 +0000 UTC m=+393.046420542"
Sep 15 18:57:10.787054 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:10.787003    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl" podStartSLOduration=1.847799231 podStartE2EDuration="5.786985276s" totalImagesPullingTime="3.939186045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:57:10.785535299 +0000 UTC m=+393.062304157" watchObservedRunningTime="2026-09-15 18:57:10.786985276 +0000 UTC m=+393.063754133"
Sep 15 18:57:11.760898 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:11.760865    2550 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-868tl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 18:57:11.761268 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:11.760927    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl" podUID="62d73c51-cdb6-453c-af2a-6f77869612b7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 18:57:14.760719 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:14.760681    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:15.669818 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:15.669786    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ncpmd"]
Sep 15 18:57:15.674194 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:15.674175    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ncpmd"
Sep 15 18:57:15.676507 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:15.676485    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-99tbh\""
Sep 15 18:57:15.676629 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:15.676512    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 15 18:57:15.682843 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:15.682819    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ncpmd"]
Sep 15 18:57:15.748027 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:15.747996    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a817227a-1d88-42ea-8e62-37bbadb82cf7-tls-certs\") pod \"model-serving-api-54d89bd5fc-ncpmd\" (UID: \"a817227a-1d88-42ea-8e62-37bbadb82cf7\") " pod="opendatahub/model-serving-api-54d89bd5fc-ncpmd"
Sep 15 18:57:15.748204 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:15.748052    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbj8q\" (UniqueName: \"kubernetes.io/projected/a817227a-1d88-42ea-8e62-37bbadb82cf7-kube-api-access-cbj8q\") pod \"model-serving-api-54d89bd5fc-ncpmd\" (UID: \"a817227a-1d88-42ea-8e62-37bbadb82cf7\") " pod="opendatahub/model-serving-api-54d89bd5fc-ncpmd"
Sep 15 18:57:15.848822 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:15.848792    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a817227a-1d88-42ea-8e62-37bbadb82cf7-tls-certs\") pod \"model-serving-api-54d89bd5fc-ncpmd\" (UID: \"a817227a-1d88-42ea-8e62-37bbadb82cf7\") " pod="opendatahub/model-serving-api-54d89bd5fc-ncpmd"
Sep 15 18:57:15.849179 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:15.848841    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbj8q\" (UniqueName: \"kubernetes.io/projected/a817227a-1d88-42ea-8e62-37bbadb82cf7-kube-api-access-cbj8q\") pod \"model-serving-api-54d89bd5fc-ncpmd\" (UID: \"a817227a-1d88-42ea-8e62-37bbadb82cf7\") " pod="opendatahub/model-serving-api-54d89bd5fc-ncpmd"
Sep 15 18:57:15.849179 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:57:15.848944    2550 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 15 18:57:15.849179 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:57:15.849021    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a817227a-1d88-42ea-8e62-37bbadb82cf7-tls-certs podName:a817227a-1d88-42ea-8e62-37bbadb82cf7 nodeName:}" failed. No retries permitted until 2026-09-15 18:57:16.349005845 +0000 UTC m=+398.625774684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a817227a-1d88-42ea-8e62-37bbadb82cf7-tls-certs") pod "model-serving-api-54d89bd5fc-ncpmd" (UID: "a817227a-1d88-42ea-8e62-37bbadb82cf7") : secret "model-serving-api-tls" not found
Sep 15 18:57:15.857617 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:15.857593    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbj8q\" (UniqueName: \"kubernetes.io/projected/a817227a-1d88-42ea-8e62-37bbadb82cf7-kube-api-access-cbj8q\") pod \"model-serving-api-54d89bd5fc-ncpmd\" (UID: \"a817227a-1d88-42ea-8e62-37bbadb82cf7\") " pod="opendatahub/model-serving-api-54d89bd5fc-ncpmd"
Sep 15 18:57:16.354241 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:16.354207    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a817227a-1d88-42ea-8e62-37bbadb82cf7-tls-certs\") pod \"model-serving-api-54d89bd5fc-ncpmd\" (UID: \"a817227a-1d88-42ea-8e62-37bbadb82cf7\") " pod="opendatahub/model-serving-api-54d89bd5fc-ncpmd"
Sep 15 18:57:16.356585 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:16.356560    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a817227a-1d88-42ea-8e62-37bbadb82cf7-tls-certs\") pod \"model-serving-api-54d89bd5fc-ncpmd\" (UID: \"a817227a-1d88-42ea-8e62-37bbadb82cf7\") " pod="opendatahub/model-serving-api-54d89bd5fc-ncpmd"
Sep 15 18:57:16.586149 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:16.586098    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ncpmd"
Sep 15 18:57:16.857609 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:16.857571    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ncpmd"]
Sep 15 18:57:16.860515 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:57:16.860488    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda817227a_1d88_42ea_8e62_37bbadb82cf7.slice/crio-79d411c5341387c9c3bbc01da16e48b1563b20b8eb0b85cd2677d4ec52a56e7a WatchSource:0}: Error finding container 79d411c5341387c9c3bbc01da16e48b1563b20b8eb0b85cd2677d4ec52a56e7a: Status 404 returned error can't find the container with id 79d411c5341387c9c3bbc01da16e48b1563b20b8eb0b85cd2677d4ec52a56e7a
Sep 15 18:57:17.779566 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:17.779530    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ncpmd" event={"ID":"a817227a-1d88-42ea-8e62-37bbadb82cf7","Type":"ContainerStarted","Data":"79d411c5341387c9c3bbc01da16e48b1563b20b8eb0b85cd2677d4ec52a56e7a"}
Sep 15 18:57:19.789537 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:19.789492    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ncpmd" event={"ID":"a817227a-1d88-42ea-8e62-37bbadb82cf7","Type":"ContainerStarted","Data":"5c1ebd7dfd4268f0a845741d351fbecf94231f5e701cf78d8167883d2d94a46d"}
Sep 15 18:57:19.790009 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:19.789682    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ncpmd"
Sep 15 18:57:19.804290 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:19.804217    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ncpmd" podStartSLOduration=2.548079565 podStartE2EDuration="4.804191343s" totalImagesPullingTime="2.256111778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:57:19.803752221 +0000 UTC m=+402.080521089" watchObservedRunningTime="2026-09-15 18:57:19.804191343 +0000 UTC m=+402.080960206"
Sep 15 18:57:24.877014 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:24.876980    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g"]
Sep 15 18:57:24.880503 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:24.880478    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g"
Sep 15 18:57:24.882938 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:24.882915    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7tlsw\""
Sep 15 18:57:24.883137 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:24.882928    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 15 18:57:24.894791 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:24.894767    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5"]
Sep 15 18:57:24.898151 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:24.898130    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g"]
Sep 15 18:57:24.898280 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:24.898262    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5"
Sep 15 18:57:24.900860 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:24.900836    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 15 18:57:24.901008 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:24.900987    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-fjsqn\""
Sep 15 18:57:24.911426 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:24.911402    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5"]
Sep 15 18:57:25.034644 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.034613    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvlt5\" (UniqueName: \"kubernetes.io/projected/a0002c4e-6120-4e46-bb2c-2362d920e0e3-kube-api-access-cvlt5\") pod \"llmisvc-controller-manager-777c6bd7d5-zg5g5\" (UID: \"a0002c4e-6120-4e46-bb2c-2362d920e0e3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5"
Sep 15 18:57:25.034644 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.034644    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0002c4e-6120-4e46-bb2c-2362d920e0e3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zg5g5\" (UID: \"a0002c4e-6120-4e46-bb2c-2362d920e0e3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5"
Sep 15 18:57:25.034851 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.034687    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85a2fbc0-1096-45fc-ac95-95f70dca3516-cert\") pod \"kserve-controller-manager-d6f4b65b8-x8g9g\" (UID: \"85a2fbc0-1096-45fc-ac95-95f70dca3516\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g"
Sep 15 18:57:25.034851 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.034749    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rkzl\" (UniqueName: \"kubernetes.io/projected/85a2fbc0-1096-45fc-ac95-95f70dca3516-kube-api-access-4rkzl\") pod \"kserve-controller-manager-d6f4b65b8-x8g9g\" (UID: \"85a2fbc0-1096-45fc-ac95-95f70dca3516\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g"
Sep 15 18:57:25.135897 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.135815    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rkzl\" (UniqueName: \"kubernetes.io/projected/85a2fbc0-1096-45fc-ac95-95f70dca3516-kube-api-access-4rkzl\") pod \"kserve-controller-manager-d6f4b65b8-x8g9g\" (UID: \"85a2fbc0-1096-45fc-ac95-95f70dca3516\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g"
Sep 15 18:57:25.135897 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.135850    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvlt5\" (UniqueName: \"kubernetes.io/projected/a0002c4e-6120-4e46-bb2c-2362d920e0e3-kube-api-access-cvlt5\") pod \"llmisvc-controller-manager-777c6bd7d5-zg5g5\" (UID: \"a0002c4e-6120-4e46-bb2c-2362d920e0e3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5"
Sep 15 18:57:25.135897 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.135871    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0002c4e-6120-4e46-bb2c-2362d920e0e3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zg5g5\" (UID: \"a0002c4e-6120-4e46-bb2c-2362d920e0e3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5"
Sep 15 18:57:25.136120 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.135912    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85a2fbc0-1096-45fc-ac95-95f70dca3516-cert\") pod \"kserve-controller-manager-d6f4b65b8-x8g9g\" (UID: \"85a2fbc0-1096-45fc-ac95-95f70dca3516\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g"
Sep 15 18:57:25.136120 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:57:25.136020    2550 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 15 18:57:25.136120 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:57:25.136022    2550 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 15 18:57:25.136120 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:57:25.136093    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85a2fbc0-1096-45fc-ac95-95f70dca3516-cert podName:85a2fbc0-1096-45fc-ac95-95f70dca3516 nodeName:}" failed. No retries permitted until 2026-09-15 18:57:25.636079092 +0000 UTC m=+407.912847925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/85a2fbc0-1096-45fc-ac95-95f70dca3516-cert") pod "kserve-controller-manager-d6f4b65b8-x8g9g" (UID: "85a2fbc0-1096-45fc-ac95-95f70dca3516") : secret "kserve-webhook-server-cert" not found
Sep 15 18:57:25.136120 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:57:25.136106    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0002c4e-6120-4e46-bb2c-2362d920e0e3-cert podName:a0002c4e-6120-4e46-bb2c-2362d920e0e3 nodeName:}" failed. No retries permitted until 2026-09-15 18:57:25.636100429 +0000 UTC m=+407.912869262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a0002c4e-6120-4e46-bb2c-2362d920e0e3-cert") pod "llmisvc-controller-manager-777c6bd7d5-zg5g5" (UID: "a0002c4e-6120-4e46-bb2c-2362d920e0e3") : secret "llmisvc-webhook-server-cert" not found
Sep 15 18:57:25.162080 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.162049    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rkzl\" (UniqueName: \"kubernetes.io/projected/85a2fbc0-1096-45fc-ac95-95f70dca3516-kube-api-access-4rkzl\") pod \"kserve-controller-manager-d6f4b65b8-x8g9g\" (UID: \"85a2fbc0-1096-45fc-ac95-95f70dca3516\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g"
Sep 15 18:57:25.162550 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.162528    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvlt5\" (UniqueName: \"kubernetes.io/projected/a0002c4e-6120-4e46-bb2c-2362d920e0e3-kube-api-access-cvlt5\") pod \"llmisvc-controller-manager-777c6bd7d5-zg5g5\" (UID: \"a0002c4e-6120-4e46-bb2c-2362d920e0e3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5"
Sep 15 18:57:25.639272 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.639237    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0002c4e-6120-4e46-bb2c-2362d920e0e3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zg5g5\" (UID: \"a0002c4e-6120-4e46-bb2c-2362d920e0e3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5"
Sep 15 18:57:25.639457 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.639392    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85a2fbc0-1096-45fc-ac95-95f70dca3516-cert\") pod \"kserve-controller-manager-d6f4b65b8-x8g9g\" (UID: \"85a2fbc0-1096-45fc-ac95-95f70dca3516\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g"
Sep 15 18:57:25.641563 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.641529    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0002c4e-6120-4e46-bb2c-2362d920e0e3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zg5g5\" (UID: \"a0002c4e-6120-4e46-bb2c-2362d920e0e3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5"
Sep 15 18:57:25.641682 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.641598    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85a2fbc0-1096-45fc-ac95-95f70dca3516-cert\") pod \"kserve-controller-manager-d6f4b65b8-x8g9g\" (UID: \"85a2fbc0-1096-45fc-ac95-95f70dca3516\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g"
Sep 15 18:57:25.793933 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.793901    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g"
Sep 15 18:57:25.810380 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:25.810354    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5"
Sep 15 18:57:26.065637 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:26.065614    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g"]
Sep 15 18:57:26.067591 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:57:26.067561    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a2fbc0_1096_45fc_ac95_95f70dca3516.slice/crio-d9bff60d5dc4c429dd01d68b6f7580a1f0d9f3dbb254cbaa8910f7f9711307db WatchSource:0}: Error finding container d9bff60d5dc4c429dd01d68b6f7580a1f0d9f3dbb254cbaa8910f7f9711307db: Status 404 returned error can't find the container with id d9bff60d5dc4c429dd01d68b6f7580a1f0d9f3dbb254cbaa8910f7f9711307db
Sep 15 18:57:26.088461 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:26.088435    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5"]
Sep 15 18:57:26.090886 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:57:26.090861    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda0002c4e_6120_4e46_bb2c_2362d920e0e3.slice/crio-c8a614f6885da69547d51490b4cebcb7f8a25054e1ae2af09ffe213b85cc1209 WatchSource:0}: Error finding container c8a614f6885da69547d51490b4cebcb7f8a25054e1ae2af09ffe213b85cc1209: Status 404 returned error can't find the container with id c8a614f6885da69547d51490b4cebcb7f8a25054e1ae2af09ffe213b85cc1209
Sep 15 18:57:26.823749 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:26.823718    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g" event={"ID":"85a2fbc0-1096-45fc-ac95-95f70dca3516","Type":"ContainerStarted","Data":"d9bff60d5dc4c429dd01d68b6f7580a1f0d9f3dbb254cbaa8910f7f9711307db"}
Sep 15 18:57:26.825027 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:26.825000    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5" event={"ID":"a0002c4e-6120-4e46-bb2c-2362d920e0e3","Type":"ContainerStarted","Data":"c8a614f6885da69547d51490b4cebcb7f8a25054e1ae2af09ffe213b85cc1209"}
Sep 15 18:57:27.524011 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.523976    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"]
Sep 15 18:57:27.524470 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.524203    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl" podUID="62d73c51-cdb6-453c-af2a-6f77869612b7" containerName="discovery" containerID="cri-o://8f5b71588738a621926bfc097162deb03c31a119151fb0f6b5264490a4a4de79" gracePeriod=30
Sep 15 18:57:27.813176 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.813139    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:27.833280 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.831011    2550 generic.go:356] "Generic (PLEG): container finished" podID="62d73c51-cdb6-453c-af2a-6f77869612b7" containerID="8f5b71588738a621926bfc097162deb03c31a119151fb0f6b5264490a4a4de79" exitCode=0
Sep 15 18:57:27.833280 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.831114    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl" event={"ID":"62d73c51-cdb6-453c-af2a-6f77869612b7","Type":"ContainerDied","Data":"8f5b71588738a621926bfc097162deb03c31a119151fb0f6b5264490a4a4de79"}
Sep 15 18:57:27.833280 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.831191    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl" event={"ID":"62d73c51-cdb6-453c-af2a-6f77869612b7","Type":"ContainerDied","Data":"454944c8c9fbb569d967c5b322670097d10da878dc31de8ec91f4d0b19607120"}
Sep 15 18:57:27.833280 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.831217    2550 scope.go:117] "RemoveContainer" containerID="8f5b71588738a621926bfc097162deb03c31a119151fb0f6b5264490a4a4de79"
Sep 15 18:57:27.833280 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.831391    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"
Sep 15 18:57:27.846350 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.846303    2550 scope.go:117] "RemoveContainer" containerID="8f5b71588738a621926bfc097162deb03c31a119151fb0f6b5264490a4a4de79"
Sep 15 18:57:27.846691 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:57:27.846663    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5b71588738a621926bfc097162deb03c31a119151fb0f6b5264490a4a4de79\": container with ID starting with 8f5b71588738a621926bfc097162deb03c31a119151fb0f6b5264490a4a4de79 not found: ID does not exist" containerID="8f5b71588738a621926bfc097162deb03c31a119151fb0f6b5264490a4a4de79"
Sep 15 18:57:27.846777 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.846718    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5b71588738a621926bfc097162deb03c31a119151fb0f6b5264490a4a4de79"} err="failed to get container status \"8f5b71588738a621926bfc097162deb03c31a119151fb0f6b5264490a4a4de79\": rpc error: code = NotFound desc = could not find container \"8f5b71588738a621926bfc097162deb03c31a119151fb0f6b5264490a4a4de79\": container with ID starting with 8f5b71588738a621926bfc097162deb03c31a119151fb0f6b5264490a4a4de79 not found: ID does not exist"
Sep 15 18:57:27.961323 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.961282    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-csr-dns-cert\") pod \"62d73c51-cdb6-453c-af2a-6f77869612b7\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") "
Sep 15 18:57:27.961469 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.961366    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-token\") pod \"62d73c51-cdb6-453c-af2a-6f77869612b7\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") "
Sep 15 18:57:27.961469 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.961403    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-kubeconfig\") pod \"62d73c51-cdb6-453c-af2a-6f77869612b7\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") "
Sep 15 18:57:27.961469 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.961432    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d73c51-cdb6-453c-af2a-6f77869612b7-local-certs\") pod \"62d73c51-cdb6-453c-af2a-6f77869612b7\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") "
Sep 15 18:57:27.961592 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.961475    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-csr-ca-configmap\") pod \"62d73c51-cdb6-453c-af2a-6f77869612b7\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") "
Sep 15 18:57:27.961592 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.961511    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kc4q\" (UniqueName: \"kubernetes.io/projected/62d73c51-cdb6-453c-af2a-6f77869612b7-kube-api-access-4kc4q\") pod \"62d73c51-cdb6-453c-af2a-6f77869612b7\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") "
Sep 15 18:57:27.961592 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.961583    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-cacerts\") pod \"62d73c51-cdb6-453c-af2a-6f77869612b7\" (UID: \"62d73c51-cdb6-453c-af2a-6f77869612b7\") "
Sep 15 18:57:27.961954 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.961905    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "62d73c51-cdb6-453c-af2a-6f77869612b7" (UID: "62d73c51-cdb6-453c-af2a-6f77869612b7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:57:27.963754 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.963725    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "62d73c51-cdb6-453c-af2a-6f77869612b7" (UID: "62d73c51-cdb6-453c-af2a-6f77869612b7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:57:27.963969 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.963943    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-cacerts" (OuterVolumeSpecName: "cacerts") pod "62d73c51-cdb6-453c-af2a-6f77869612b7" (UID: "62d73c51-cdb6-453c-af2a-6f77869612b7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:57:27.964052 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.963944    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "62d73c51-cdb6-453c-af2a-6f77869612b7" (UID: "62d73c51-cdb6-453c-af2a-6f77869612b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:57:27.964128 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.964104    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "62d73c51-cdb6-453c-af2a-6f77869612b7" (UID: "62d73c51-cdb6-453c-af2a-6f77869612b7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:57:27.964269 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.964246    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d73c51-cdb6-453c-af2a-6f77869612b7-local-certs" (OuterVolumeSpecName: "local-certs") pod "62d73c51-cdb6-453c-af2a-6f77869612b7" (UID: "62d73c51-cdb6-453c-af2a-6f77869612b7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:57:27.964389 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:27.964277    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d73c51-cdb6-453c-af2a-6f77869612b7-kube-api-access-4kc4q" (OuterVolumeSpecName: "kube-api-access-4kc4q") pod "62d73c51-cdb6-453c-af2a-6f77869612b7" (UID: "62d73c51-cdb6-453c-af2a-6f77869612b7"). InnerVolumeSpecName "kube-api-access-4kc4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:57:28.063255 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:28.063182    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kc4q\" (UniqueName: \"kubernetes.io/projected/62d73c51-cdb6-453c-af2a-6f77869612b7-kube-api-access-4kc4q\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:57:28.063255 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:28.063210    2550 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-cacerts\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:57:28.063255 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:28.063220    2550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-csr-dns-cert\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:57:28.063255 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:28.063232    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-token\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:57:28.063255 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:28.063246    2550 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-kubeconfig\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:57:28.063255 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:28.063258    2550 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d73c51-cdb6-453c-af2a-6f77869612b7-local-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:57:28.063594 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:28.063271    2550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/62d73c51-cdb6-453c-af2a-6f77869612b7-istio-csr-ca-configmap\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:57:28.157707 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:28.157672    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"]
Sep 15 18:57:28.162340 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:28.162301    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-868tl"]
Sep 15 18:57:28.295358 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:28.295322    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62d73c51-cdb6-453c-af2a-6f77869612b7" path="/var/lib/kubelet/pods/62d73c51-cdb6-453c-af2a-6f77869612b7/volumes"
Sep 15 18:57:28.836820 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:28.836723    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g" event={"ID":"85a2fbc0-1096-45fc-ac95-95f70dca3516","Type":"ContainerStarted","Data":"fedc4fb25f60ecdfef40fbd3956398b91dafd5e683cca6e6d28c7b5e46812fd7"}
Sep 15 18:57:28.837237 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:28.836827    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g"
Sep 15 18:57:28.852870 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:28.852817    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g" podStartSLOduration=2.368956071 podStartE2EDuration="4.852798787s" totalImagesPullingTime="2.483842716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:57:28.850924019 +0000 UTC m=+411.127692875" watchObservedRunningTime="2026-09-15 18:57:28.852798787 +0000 UTC m=+411.129567643"
Sep 15 18:57:29.844257 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:29.844225    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5" event={"ID":"a0002c4e-6120-4e46-bb2c-2362d920e0e3","Type":"ContainerStarted","Data":"85d9e5f369f6d3de33adc54c7a59ed4d0583716ad1c004f66f7fe5b2c7946dd6"}
Sep 15 18:57:29.844735 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:29.844509    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5"
Sep 15 18:57:29.860055 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:29.859889    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5" podStartSLOduration=2.620799364 podStartE2EDuration="5.859876906s" totalImagesPullingTime="3.239077542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:57:29.858897293 +0000 UTC m=+412.135666149" watchObservedRunningTime="2026-09-15 18:57:29.859876906 +0000 UTC m=+412.136645762"
Sep 15 18:57:30.797125 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:30.797097    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ncpmd"
Sep 15 18:57:59.849556 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:57:59.849526    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x8g9g"
Sep 15 18:58:00.850807 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:00.850739    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zg5g5"
Sep 15 18:58:32.668361 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:32.668324    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-fmrzn"]
Sep 15 18:58:32.668862 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:32.668616    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d73c51-cdb6-453c-af2a-6f77869612b7" containerName="discovery"
Sep 15 18:58:32.668862 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:32.668627    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d73c51-cdb6-453c-af2a-6f77869612b7" containerName="discovery"
Sep 15 18:58:32.668862 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:32.668678    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d73c51-cdb6-453c-af2a-6f77869612b7" containerName="discovery"
Sep 15 18:58:32.674650 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:32.674624    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-fmrzn"
Sep 15 18:58:32.676802 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:32.676778    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 18:58:32.676924 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:32.676802    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xcwnx\""
Sep 15 18:58:32.676924 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:32.676820    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 18:58:32.679659 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:32.679636    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-fmrzn"]
Sep 15 18:58:32.751006 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:32.750971    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x7dk\" (UniqueName: \"kubernetes.io/projected/8f3b364f-b9f7-4642-91a5-0da94194e3b2-kube-api-access-7x7dk\") pod \"authorino-operator-76b478bd78-fmrzn\" (UID: \"8f3b364f-b9f7-4642-91a5-0da94194e3b2\") " pod="kuadrant-system/authorino-operator-76b478bd78-fmrzn"
Sep 15 18:58:32.851934 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:32.851897    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x7dk\" (UniqueName: \"kubernetes.io/projected/8f3b364f-b9f7-4642-91a5-0da94194e3b2-kube-api-access-7x7dk\") pod \"authorino-operator-76b478bd78-fmrzn\" (UID: \"8f3b364f-b9f7-4642-91a5-0da94194e3b2\") " pod="kuadrant-system/authorino-operator-76b478bd78-fmrzn"
Sep 15 18:58:32.868342 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:32.868302    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x7dk\" (UniqueName: \"kubernetes.io/projected/8f3b364f-b9f7-4642-91a5-0da94194e3b2-kube-api-access-7x7dk\") pod \"authorino-operator-76b478bd78-fmrzn\" (UID: \"8f3b364f-b9f7-4642-91a5-0da94194e3b2\") " pod="kuadrant-system/authorino-operator-76b478bd78-fmrzn"
Sep 15 18:58:32.985569 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:32.985475    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-fmrzn"
Sep 15 18:58:33.259718 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:33.259640    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-fmrzn"]
Sep 15 18:58:33.262923 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:58:33.262893    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3b364f_b9f7_4642_91a5_0da94194e3b2.slice/crio-2aef1ced69dedc18012a435a2c684209862d99310e38474f534c50a02c6d6934 WatchSource:0}: Error finding container 2aef1ced69dedc18012a435a2c684209862d99310e38474f534c50a02c6d6934: Status 404 returned error can't find the container with id 2aef1ced69dedc18012a435a2c684209862d99310e38474f534c50a02c6d6934
Sep 15 18:58:34.085185 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:34.085149    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-fmrzn" event={"ID":"8f3b364f-b9f7-4642-91a5-0da94194e3b2","Type":"ContainerStarted","Data":"2aef1ced69dedc18012a435a2c684209862d99310e38474f534c50a02c6d6934"}
Sep 15 18:58:36.094909 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:36.094825    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-fmrzn" event={"ID":"8f3b364f-b9f7-4642-91a5-0da94194e3b2","Type":"ContainerStarted","Data":"63eb6592507d5264a8a0d8a95c9cc0e2706199501f15ae9b411cc41f826e1a73"}
Sep 15 18:58:36.095256 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:36.095074    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-fmrzn"
Sep 15 18:58:36.111742 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:36.111684    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-fmrzn" podStartSLOduration=1.5983401069999998 podStartE2EDuration="4.111665935s" totalImagesPullingTime="2.513325828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:58:36.111586334 +0000 UTC m=+478.388355191" watchObservedRunningTime="2026-09-15 18:58:36.111665935 +0000 UTC m=+478.388434791"
Sep 15 18:58:36.793507 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:36.793472    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"]
Sep 15 18:58:36.796650 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:36.796629    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:36.799271 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:36.799250    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lhsr2\""
Sep 15 18:58:36.809674 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:36.809650    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"]
Sep 15 18:58:36.888614 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:36.888586    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\" (UID: \"2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:36.888744 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:36.888651    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmntp\" (UniqueName: \"kubernetes.io/projected/2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c-kube-api-access-cmntp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\" (UID: \"2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:36.989449 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:36.989412    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmntp\" (UniqueName: \"kubernetes.io/projected/2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c-kube-api-access-cmntp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\" (UID: \"2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:36.989449 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:36.989455    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\" (UID: \"2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:36.989792 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:36.989775    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\" (UID: \"2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:37.003428 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:37.003405    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmntp\" (UniqueName: \"kubernetes.io/projected/2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c-kube-api-access-cmntp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\" (UID: \"2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:37.105749 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:37.105670    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:37.390338 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:37.390236    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"]
Sep 15 18:58:37.394060 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:58:37.394025    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3f5f9d_c0df_49a8_a2f8_b90a28f2290c.slice/crio-67549fc49087bc77d53dc8042ebd53b8864bbdaf598d2d34909679732c7f65cd WatchSource:0}: Error finding container 67549fc49087bc77d53dc8042ebd53b8864bbdaf598d2d34909679732c7f65cd: Status 404 returned error can't find the container with id 67549fc49087bc77d53dc8042ebd53b8864bbdaf598d2d34909679732c7f65cd
Sep 15 18:58:38.107767 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:38.107726    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4" event={"ID":"2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c","Type":"ContainerStarted","Data":"67549fc49087bc77d53dc8042ebd53b8864bbdaf598d2d34909679732c7f65cd"}
Sep 15 18:58:38.805570 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:38.805524    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"]
Sep 15 18:58:38.808930 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:38.808909    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:38.811821 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:38.811798    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-m6mgh\""
Sep 15 18:58:38.826066 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:38.826040    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"]
Sep 15 18:58:38.907996 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:38.907958    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snq7c\" (UniqueName: \"kubernetes.io/projected/c744fdb1-3add-458f-9885-d9a8e9f9c295-kube-api-access-snq7c\") pod \"limitador-operator-controller-manager-74495c69f8-kw9rl\" (UID: \"c744fdb1-3add-458f-9885-d9a8e9f9c295\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:39.009307 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:39.009260    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snq7c\" (UniqueName: \"kubernetes.io/projected/c744fdb1-3add-458f-9885-d9a8e9f9c295-kube-api-access-snq7c\") pod \"limitador-operator-controller-manager-74495c69f8-kw9rl\" (UID: \"c744fdb1-3add-458f-9885-d9a8e9f9c295\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:39.023409 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:39.023376    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snq7c\" (UniqueName: \"kubernetes.io/projected/c744fdb1-3add-458f-9885-d9a8e9f9c295-kube-api-access-snq7c\") pod \"limitador-operator-controller-manager-74495c69f8-kw9rl\" (UID: \"c744fdb1-3add-458f-9885-d9a8e9f9c295\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:39.120852 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:39.120776    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:39.403096 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:39.403018    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"]
Sep 15 18:58:39.406997 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:58:39.406962    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc744fdb1_3add_458f_9885_d9a8e9f9c295.slice/crio-3172d6993582db9cb379502761cefe124af477c50e5a85476c0d8bc7b9224959 WatchSource:0}: Error finding container 3172d6993582db9cb379502761cefe124af477c50e5a85476c0d8bc7b9224959: Status 404 returned error can't find the container with id 3172d6993582db9cb379502761cefe124af477c50e5a85476c0d8bc7b9224959
Sep 15 18:58:40.118981 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:40.118943    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl" event={"ID":"c744fdb1-3add-458f-9885-d9a8e9f9c295","Type":"ContainerStarted","Data":"3172d6993582db9cb379502761cefe124af477c50e5a85476c0d8bc7b9224959"}
Sep 15 18:58:41.125641 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:41.125598    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4" event={"ID":"2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c","Type":"ContainerStarted","Data":"f6795f28c45b60b5c2526b0bf57944f86ef10c9f9ed19682791c8a3048b06f51"}
Sep 15 18:58:41.126095 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:41.125704    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:41.154218 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:41.154130    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4" podStartSLOduration=1.549055681 podStartE2EDuration="5.154111199s" totalImagesPullingTime="3.605055518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:58:41.151419047 +0000 UTC m=+483.428187902" watchObservedRunningTime="2026-09-15 18:58:41.154111199 +0000 UTC m=+483.430880055"
Sep 15 18:58:42.131369 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:42.131278    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl" event={"ID":"c744fdb1-3add-458f-9885-d9a8e9f9c295","Type":"ContainerStarted","Data":"877c89adf0fee4dadb02e30a74ac5366ecd23b6b3262e3095f48a5628d074a8a"}
Sep 15 18:58:42.152898 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:42.152847    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl" podStartSLOduration=1.7525479320000001 podStartE2EDuration="4.152832038s" totalImagesPullingTime="2.400284106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:58:42.152219604 +0000 UTC m=+484.428988485" watchObservedRunningTime="2026-09-15 18:58:42.152832038 +0000 UTC m=+484.429600894"
Sep 15 18:58:43.135852 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:43.135823    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:47.100494 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:47.100465    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-fmrzn"
Sep 15 18:58:52.134001 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:52.133966    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:54.142709 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.142677    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:54.377085 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.377051    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"]
Sep 15 18:58:54.377246 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.377229    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4" podUID="2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" containerName="manager" containerID="cri-o://f6795f28c45b60b5c2526b0bf57944f86ef10c9f9ed19682791c8a3048b06f51" gracePeriod=2
Sep 15 18:58:54.386333 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.386287    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"]
Sep 15 18:58:54.397062 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:54.396996    2550 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\": pods \"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:54.399301 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.399278    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"]
Sep 15 18:58:54.399581 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.399553    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl" podUID="c744fdb1-3add-458f-9885-d9a8e9f9c295" containerName="manager" containerID="cri-o://877c89adf0fee4dadb02e30a74ac5366ecd23b6b3262e3095f48a5628d074a8a" gracePeriod=2
Sep 15 18:58:54.410682 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.410658    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"]
Sep 15 18:58:54.410978 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:54.410946    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:54.411141 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.411122    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" containerName="manager"
Sep 15 18:58:54.411202 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.411144    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" containerName="manager"
Sep 15 18:58:54.411305 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.411217    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" containerName="manager"
Sep 15 18:58:54.419655 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.417637    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"]
Sep 15 18:58:54.419655 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.417756    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"
Sep 15 18:58:54.420165 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:54.420133    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:54.424738 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.424716    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-878xj"]
Sep 15 18:58:54.424976 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.424963    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c744fdb1-3add-458f-9885-d9a8e9f9c295" containerName="manager"
Sep 15 18:58:54.424976 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.424976    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c744fdb1-3add-458f-9885-d9a8e9f9c295" containerName="manager"
Sep 15 18:58:54.425102 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.425035    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c744fdb1-3add-458f-9885-d9a8e9f9c295" containerName="manager"
Sep 15 18:58:54.427592 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.427573    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-878xj"
Sep 15 18:58:54.428495 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.428476    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"]
Sep 15 18:58:54.442073 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.442051    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-878xj"]
Sep 15 18:58:54.448181 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:54.448145    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kw9rl\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="c744fdb1-3add-458f-9885-d9a8e9f9c295" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:54.450084 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:54.450057    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kw9rl\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="c744fdb1-3add-458f-9885-d9a8e9f9c295" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:54.451779 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:54.451755    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:54.531461 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.531435    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c11cab9b-8a4b-48f0-a233-981bf4411508-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2x5kx\" (UID: \"c11cab9b-8a4b-48f0-a233-981bf4411508\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"
Sep 15 18:58:54.531588 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.531471    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsnn\" (UniqueName: \"kubernetes.io/projected/05f61967-56f3-43eb-a1b9-891c7bb05b6d-kube-api-access-nhsnn\") pod \"limitador-operator-controller-manager-74495c69f8-878xj\" (UID: \"05f61967-56f3-43eb-a1b9-891c7bb05b6d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-878xj"
Sep 15 18:58:54.531658 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.531604    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh2xz\" (UniqueName: \"kubernetes.io/projected/c11cab9b-8a4b-48f0-a233-981bf4411508-kube-api-access-jh2xz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2x5kx\" (UID: \"c11cab9b-8a4b-48f0-a233-981bf4411508\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"
Sep 15 18:58:54.632768 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.632740    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2xz\" (UniqueName: \"kubernetes.io/projected/c11cab9b-8a4b-48f0-a233-981bf4411508-kube-api-access-jh2xz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2x5kx\" (UID: \"c11cab9b-8a4b-48f0-a233-981bf4411508\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"
Sep 15 18:58:54.632877 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.632804    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c11cab9b-8a4b-48f0-a233-981bf4411508-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2x5kx\" (UID: \"c11cab9b-8a4b-48f0-a233-981bf4411508\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"
Sep 15 18:58:54.632877 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.632842    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsnn\" (UniqueName: \"kubernetes.io/projected/05f61967-56f3-43eb-a1b9-891c7bb05b6d-kube-api-access-nhsnn\") pod \"limitador-operator-controller-manager-74495c69f8-878xj\" (UID: \"05f61967-56f3-43eb-a1b9-891c7bb05b6d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-878xj"
Sep 15 18:58:54.633266 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.633245    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c11cab9b-8a4b-48f0-a233-981bf4411508-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2x5kx\" (UID: \"c11cab9b-8a4b-48f0-a233-981bf4411508\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"
Sep 15 18:58:54.637748 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.637728    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:54.639766 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.639749    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:54.641429 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:54.641400    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:54.643809 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:54.643785    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kw9rl\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="c744fdb1-3add-458f-9885-d9a8e9f9c295" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:54.650382 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.650343    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsnn\" (UniqueName: \"kubernetes.io/projected/05f61967-56f3-43eb-a1b9-891c7bb05b6d-kube-api-access-nhsnn\") pod \"limitador-operator-controller-manager-74495c69f8-878xj\" (UID: \"05f61967-56f3-43eb-a1b9-891c7bb05b6d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-878xj"
Sep 15 18:58:54.652017 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:54.651992    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kw9rl\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="c744fdb1-3add-458f-9885-d9a8e9f9c295" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:54.653937 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.653916    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2xz\" (UniqueName: \"kubernetes.io/projected/c11cab9b-8a4b-48f0-a233-981bf4411508-kube-api-access-jh2xz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2x5kx\" (UID: \"c11cab9b-8a4b-48f0-a233-981bf4411508\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"
Sep 15 18:58:54.654103 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:54.654083    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:54.733454 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.733427    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c-extensions-socket-volume\") pod \"2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c\" (UID: \"2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c\") "
Sep 15 18:58:54.733563 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.733496    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmntp\" (UniqueName: \"kubernetes.io/projected/2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c-kube-api-access-cmntp\") pod \"2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c\" (UID: \"2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c\") "
Sep 15 18:58:54.733563 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.733527    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snq7c\" (UniqueName: \"kubernetes.io/projected/c744fdb1-3add-458f-9885-d9a8e9f9c295-kube-api-access-snq7c\") pod \"c744fdb1-3add-458f-9885-d9a8e9f9c295\" (UID: \"c744fdb1-3add-458f-9885-d9a8e9f9c295\") "
Sep 15 18:58:54.733924 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.733899    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" (UID: "2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:58:54.735547 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.735525    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c-kube-api-access-cmntp" (OuterVolumeSpecName: "kube-api-access-cmntp") pod "2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" (UID: "2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c"). InnerVolumeSpecName "kube-api-access-cmntp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:58:54.735608 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.735545    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c744fdb1-3add-458f-9885-d9a8e9f9c295-kube-api-access-snq7c" (OuterVolumeSpecName: "kube-api-access-snq7c") pod "c744fdb1-3add-458f-9885-d9a8e9f9c295" (UID: "c744fdb1-3add-458f-9885-d9a8e9f9c295"). InnerVolumeSpecName "kube-api-access-snq7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:58:54.824901 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.824868    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"
Sep 15 18:58:54.830900 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.830884    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-878xj"
Sep 15 18:58:54.834917 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.834890    2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c-extensions-socket-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:58:54.834917 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.834913    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmntp\" (UniqueName: \"kubernetes.io/projected/2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c-kube-api-access-cmntp\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:58:54.835105 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:54.834928    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snq7c\" (UniqueName: \"kubernetes.io/projected/c744fdb1-3add-458f-9885-d9a8e9f9c295-kube-api-access-snq7c\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:58:55.103775 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.103740    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"]
Sep 15 18:58:55.105105 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.105081    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-878xj"]
Sep 15 18:58:55.107105 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:58:55.107083    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11cab9b_8a4b_48f0_a233_981bf4411508.slice/crio-02817d5f5f1aafdbd51464ac9bd92e5e37f36dcf2ad7fe215d8315653c2b33d0 WatchSource:0}: Error finding container 02817d5f5f1aafdbd51464ac9bd92e5e37f36dcf2ad7fe215d8315653c2b33d0: Status 404 returned error can't find the container with id 02817d5f5f1aafdbd51464ac9bd92e5e37f36dcf2ad7fe215d8315653c2b33d0
Sep 15 18:58:55.107755 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:58:55.107735    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f61967_56f3_43eb_a1b9_891c7bb05b6d.slice/crio-6dddb2269e088466f3a420bbaf876488eac7060a14a0e817d28f4b81f7470474 WatchSource:0}: Error finding container 6dddb2269e088466f3a420bbaf876488eac7060a14a0e817d28f4b81f7470474: Status 404 returned error can't find the container with id 6dddb2269e088466f3a420bbaf876488eac7060a14a0e817d28f4b81f7470474
Sep 15 18:58:55.189035 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.189007    2550 generic.go:356] "Generic (PLEG): container finished" podID="2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" containerID="f6795f28c45b60b5c2526b0bf57944f86ef10c9f9ed19682791c8a3048b06f51" exitCode=0
Sep 15 18:58:55.189386 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.189067    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:55.189386 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.189110    2550 scope.go:117] "RemoveContainer" containerID="f6795f28c45b60b5c2526b0bf57944f86ef10c9f9ed19682791c8a3048b06f51"
Sep 15 18:58:55.191501 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.191452    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx" event={"ID":"c11cab9b-8a4b-48f0-a233-981bf4411508","Type":"ContainerStarted","Data":"18458ff1d9af85ab60b8665449d60e520e69757170eb0df5940f5a5205c10c4d"}
Sep 15 18:58:55.191594 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.191507    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx" event={"ID":"c11cab9b-8a4b-48f0-a233-981bf4411508","Type":"ContainerStarted","Data":"02817d5f5f1aafdbd51464ac9bd92e5e37f36dcf2ad7fe215d8315653c2b33d0"}
Sep 15 18:58:55.191594 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.191537    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"
Sep 15 18:58:55.191594 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:55.191559    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:55.193130 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.193106    2550 generic.go:356] "Generic (PLEG): container finished" podID="c744fdb1-3add-458f-9885-d9a8e9f9c295" containerID="877c89adf0fee4dadb02e30a74ac5366ecd23b6b3262e3095f48a5628d074a8a" exitCode=0
Sep 15 18:58:55.193242 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.193152    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:55.195677 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.195632    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-878xj" event={"ID":"05f61967-56f3-43eb-a1b9-891c7bb05b6d","Type":"ContainerStarted","Data":"e754dab27dabca108f40a3ec6a654babf38b2620f687a57b7e25d44ad5139b4f"}
Sep 15 18:58:55.195774 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.195677    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-878xj" event={"ID":"05f61967-56f3-43eb-a1b9-891c7bb05b6d","Type":"ContainerStarted","Data":"6dddb2269e088466f3a420bbaf876488eac7060a14a0e817d28f4b81f7470474"}
Sep 15 18:58:55.195774 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.195708    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-878xj"
Sep 15 18:58:55.197432 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:55.197292    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kw9rl\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="c744fdb1-3add-458f-9885-d9a8e9f9c295" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:55.198949 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.198928    2550 scope.go:117] "RemoveContainer" containerID="f6795f28c45b60b5c2526b0bf57944f86ef10c9f9ed19682791c8a3048b06f51"
Sep 15 18:58:55.199343 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:55.199305    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6795f28c45b60b5c2526b0bf57944f86ef10c9f9ed19682791c8a3048b06f51\": container with ID starting with f6795f28c45b60b5c2526b0bf57944f86ef10c9f9ed19682791c8a3048b06f51 not found: ID does not exist" containerID="f6795f28c45b60b5c2526b0bf57944f86ef10c9f9ed19682791c8a3048b06f51"
Sep 15 18:58:55.199438 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.199352    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6795f28c45b60b5c2526b0bf57944f86ef10c9f9ed19682791c8a3048b06f51"} err="failed to get container status \"f6795f28c45b60b5c2526b0bf57944f86ef10c9f9ed19682791c8a3048b06f51\": rpc error: code = NotFound desc = could not find container \"f6795f28c45b60b5c2526b0bf57944f86ef10c9f9ed19682791c8a3048b06f51\": container with ID starting with f6795f28c45b60b5c2526b0bf57944f86ef10c9f9ed19682791c8a3048b06f51 not found: ID does not exist"
Sep 15 18:58:55.199438 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.199368    2550 scope.go:117] "RemoveContainer" containerID="877c89adf0fee4dadb02e30a74ac5366ecd23b6b3262e3095f48a5628d074a8a"
Sep 15 18:58:55.206100 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.206085    2550 scope.go:117] "RemoveContainer" containerID="877c89adf0fee4dadb02e30a74ac5366ecd23b6b3262e3095f48a5628d074a8a"
Sep 15 18:58:55.206385 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:55.206363    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"877c89adf0fee4dadb02e30a74ac5366ecd23b6b3262e3095f48a5628d074a8a\": container with ID starting with 877c89adf0fee4dadb02e30a74ac5366ecd23b6b3262e3095f48a5628d074a8a not found: ID does not exist" containerID="877c89adf0fee4dadb02e30a74ac5366ecd23b6b3262e3095f48a5628d074a8a"
Sep 15 18:58:55.206440 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.206395    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"877c89adf0fee4dadb02e30a74ac5366ecd23b6b3262e3095f48a5628d074a8a"} err="failed to get container status \"877c89adf0fee4dadb02e30a74ac5366ecd23b6b3262e3095f48a5628d074a8a\": rpc error: code = NotFound desc = could not find container \"877c89adf0fee4dadb02e30a74ac5366ecd23b6b3262e3095f48a5628d074a8a\": container with ID starting with 877c89adf0fee4dadb02e30a74ac5366ecd23b6b3262e3095f48a5628d074a8a not found: ID does not exist"
Sep 15 18:58:55.244730 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.244687    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-878xj" podStartSLOduration=1.244674252 podStartE2EDuration="1.244674252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:58:55.242893677 +0000 UTC m=+497.519662533" watchObservedRunningTime="2026-09-15 18:58:55.244674252 +0000 UTC m=+497.521443108"
Sep 15 18:58:55.246796 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:55.246776    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kw9rl\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="c744fdb1-3add-458f-9885-d9a8e9f9c295" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:55.251357 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:55.251333    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:55.283966 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:55.283933    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5s5b4\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5s5b4"
Sep 15 18:58:55.284166 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:55.284116    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx" podStartSLOduration=1.284098487 podStartE2EDuration="1.284098487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:58:55.281632219 +0000 UTC m=+497.558401090" watchObservedRunningTime="2026-09-15 18:58:55.284098487 +0000 UTC m=+497.560867409"
Sep 15 18:58:55.286163 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:58:55.286135    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kw9rl\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="c744fdb1-3add-458f-9885-d9a8e9f9c295" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kw9rl"
Sep 15 18:58:56.294110 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:56.294080    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c" path="/var/lib/kubelet/pods/2c3f5f9d-c0df-49a8-a2f8-b90a28f2290c/volumes"
Sep 15 18:58:56.294462 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:58:56.294393    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c744fdb1-3add-458f-9885-d9a8e9f9c295" path="/var/lib/kubelet/pods/c744fdb1-3add-458f-9885-d9a8e9f9c295/volumes"
Sep 15 18:59:06.204086 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:06.204053    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"
Sep 15 18:59:06.204685 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:06.204133    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-878xj"
Sep 15 18:59:11.366937 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:11.366902    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"]
Sep 15 18:59:11.367330 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:11.367121    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx" podUID="c11cab9b-8a4b-48f0-a233-981bf4411508" containerName="manager" containerID="cri-o://18458ff1d9af85ab60b8665449d60e520e69757170eb0df5940f5a5205c10c4d" gracePeriod=10
Sep 15 18:59:11.610594 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:11.610569    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"
Sep 15 18:59:11.654562 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:11.654487    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c11cab9b-8a4b-48f0-a233-981bf4411508-extensions-socket-volume\") pod \"c11cab9b-8a4b-48f0-a233-981bf4411508\" (UID: \"c11cab9b-8a4b-48f0-a233-981bf4411508\") "
Sep 15 18:59:11.654562 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:11.654523    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh2xz\" (UniqueName: \"kubernetes.io/projected/c11cab9b-8a4b-48f0-a233-981bf4411508-kube-api-access-jh2xz\") pod \"c11cab9b-8a4b-48f0-a233-981bf4411508\" (UID: \"c11cab9b-8a4b-48f0-a233-981bf4411508\") "
Sep 15 18:59:11.654868 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:11.654842    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11cab9b-8a4b-48f0-a233-981bf4411508-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c11cab9b-8a4b-48f0-a233-981bf4411508" (UID: "c11cab9b-8a4b-48f0-a233-981bf4411508"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:59:11.656686 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:11.656665    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11cab9b-8a4b-48f0-a233-981bf4411508-kube-api-access-jh2xz" (OuterVolumeSpecName: "kube-api-access-jh2xz") pod "c11cab9b-8a4b-48f0-a233-981bf4411508" (UID: "c11cab9b-8a4b-48f0-a233-981bf4411508"). InnerVolumeSpecName "kube-api-access-jh2xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:59:11.755328 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:11.755279    2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c11cab9b-8a4b-48f0-a233-981bf4411508-extensions-socket-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:59:11.755328 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:11.755328    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh2xz\" (UniqueName: \"kubernetes.io/projected/c11cab9b-8a4b-48f0-a233-981bf4411508-kube-api-access-jh2xz\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:59:12.280624 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:12.280587    2550 generic.go:356] "Generic (PLEG): container finished" podID="c11cab9b-8a4b-48f0-a233-981bf4411508" containerID="18458ff1d9af85ab60b8665449d60e520e69757170eb0df5940f5a5205c10c4d" exitCode=0
Sep 15 18:59:12.280803 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:12.280634    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"
Sep 15 18:59:12.280803 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:12.280659    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx" event={"ID":"c11cab9b-8a4b-48f0-a233-981bf4411508","Type":"ContainerDied","Data":"18458ff1d9af85ab60b8665449d60e520e69757170eb0df5940f5a5205c10c4d"}
Sep 15 18:59:12.280803 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:12.280708    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx" event={"ID":"c11cab9b-8a4b-48f0-a233-981bf4411508","Type":"ContainerDied","Data":"02817d5f5f1aafdbd51464ac9bd92e5e37f36dcf2ad7fe215d8315653c2b33d0"}
Sep 15 18:59:12.280803 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:12.280739    2550 scope.go:117] "RemoveContainer" containerID="18458ff1d9af85ab60b8665449d60e520e69757170eb0df5940f5a5205c10c4d"
Sep 15 18:59:12.289580 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:12.289554    2550 scope.go:117] "RemoveContainer" containerID="18458ff1d9af85ab60b8665449d60e520e69757170eb0df5940f5a5205c10c4d"
Sep 15 18:59:12.289830 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:59:12.289808    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18458ff1d9af85ab60b8665449d60e520e69757170eb0df5940f5a5205c10c4d\": container with ID starting with 18458ff1d9af85ab60b8665449d60e520e69757170eb0df5940f5a5205c10c4d not found: ID does not exist" containerID="18458ff1d9af85ab60b8665449d60e520e69757170eb0df5940f5a5205c10c4d"
Sep 15 18:59:12.289898 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:12.289842    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18458ff1d9af85ab60b8665449d60e520e69757170eb0df5940f5a5205c10c4d"} err="failed to get container status \"18458ff1d9af85ab60b8665449d60e520e69757170eb0df5940f5a5205c10c4d\": rpc error: code = NotFound desc = could not find container \"18458ff1d9af85ab60b8665449d60e520e69757170eb0df5940f5a5205c10c4d\": container with ID starting with 18458ff1d9af85ab60b8665449d60e520e69757170eb0df5940f5a5205c10c4d not found: ID does not exist"
Sep 15 18:59:12.308092 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:12.308062    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"]
Sep 15 18:59:12.317018 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:12.316994    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2x5kx"]
Sep 15 18:59:14.294269 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:14.294231    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c11cab9b-8a4b-48f0-a233-981bf4411508" path="/var/lib/kubelet/pods/c11cab9b-8a4b-48f0-a233-981bf4411508/volumes"
Sep 15 18:59:27.901520 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.901488    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"]
Sep 15 18:59:27.901957 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.901929    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c11cab9b-8a4b-48f0-a233-981bf4411508" containerName="manager"
Sep 15 18:59:27.901957 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.901947    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c11cab9b-8a4b-48f0-a233-981bf4411508" containerName="manager"
Sep 15 18:59:27.902063 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.902024    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c11cab9b-8a4b-48f0-a233-981bf4411508" containerName="manager"
Sep 15 18:59:27.904720 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.904699    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:27.907078 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.907054    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zqzmh\""
Sep 15 18:59:27.907078 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.907066    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 18:59:27.915353 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.915329    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"]
Sep 15 18:59:27.975228 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.975204    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8190ca1e-7e85-490e-b665-4712192e4d60-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:27.975382 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.975235    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8190ca1e-7e85-490e-b665-4712192e4d60-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:27.975382 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.975266    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8190ca1e-7e85-490e-b665-4712192e4d60-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:27.975382 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.975356    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8190ca1e-7e85-490e-b665-4712192e4d60-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:27.975497 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.975380    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8190ca1e-7e85-490e-b665-4712192e4d60-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:27.975497 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.975403    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b2fg\" (UniqueName: \"kubernetes.io/projected/8190ca1e-7e85-490e-b665-4712192e4d60-kube-api-access-8b2fg\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:27.975497 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.975460    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8190ca1e-7e85-490e-b665-4712192e4d60-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:27.975592 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.975495    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8190ca1e-7e85-490e-b665-4712192e4d60-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:27.975592 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:27.975510    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8190ca1e-7e85-490e-b665-4712192e4d60-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.076548 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.076523    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8190ca1e-7e85-490e-b665-4712192e4d60-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.076677 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.076553    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8190ca1e-7e85-490e-b665-4712192e4d60-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.076677 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.076582    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8190ca1e-7e85-490e-b665-4712192e4d60-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.076677 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.076599    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8190ca1e-7e85-490e-b665-4712192e4d60-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.076677 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.076620    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8190ca1e-7e85-490e-b665-4712192e4d60-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.076677 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.076650    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8190ca1e-7e85-490e-b665-4712192e4d60-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.076677 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.076675    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8190ca1e-7e85-490e-b665-4712192e4d60-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.076973 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.076701    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b2fg\" (UniqueName: \"kubernetes.io/projected/8190ca1e-7e85-490e-b665-4712192e4d60-kube-api-access-8b2fg\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.076973 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.076744    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8190ca1e-7e85-490e-b665-4712192e4d60-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.077076 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.077028    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8190ca1e-7e85-490e-b665-4712192e4d60-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.077148 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.077122    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8190ca1e-7e85-490e-b665-4712192e4d60-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.077244 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.077224    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8190ca1e-7e85-490e-b665-4712192e4d60-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.077285 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.077266    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8190ca1e-7e85-490e-b665-4712192e4d60-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.077638 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.077618    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8190ca1e-7e85-490e-b665-4712192e4d60-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.078955 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.078934    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8190ca1e-7e85-490e-b665-4712192e4d60-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.079275 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.079254    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8190ca1e-7e85-490e-b665-4712192e4d60-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.084184 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.084157    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8190ca1e-7e85-490e-b665-4712192e4d60-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.084330 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.084292    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b2fg\" (UniqueName: \"kubernetes.io/projected/8190ca1e-7e85-490e-b665-4712192e4d60-kube-api-access-8b2fg\") pod \"maas-default-gateway-openshift-default-78bb78ff78-ht96l\" (UID: \"8190ca1e-7e85-490e-b665-4712192e4d60\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.215809 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.215731    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:28.488967 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:28.488918    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"]
Sep 15 18:59:28.491078 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:59:28.491050    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8190ca1e_7e85_490e_b665_4712192e4d60.slice/crio-063622c8471e0e8c0bc33b02960f5edb3c674e825fdcacf94e345b55fecb4187 WatchSource:0}: Error finding container 063622c8471e0e8c0bc33b02960f5edb3c674e825fdcacf94e345b55fecb4187: Status 404 returned error can't find the container with id 063622c8471e0e8c0bc33b02960f5edb3c674e825fdcacf94e345b55fecb4187
Sep 15 18:59:29.354093 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:29.354055    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l" event={"ID":"8190ca1e-7e85-490e-b665-4712192e4d60","Type":"ContainerStarted","Data":"063622c8471e0e8c0bc33b02960f5edb3c674e825fdcacf94e345b55fecb4187"}
Sep 15 18:59:31.238360 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:31.238328    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 18:59:31.238611 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:31.238413    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 18:59:31.238611 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:31.238448    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 18:59:31.365517 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:31.365479    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l" event={"ID":"8190ca1e-7e85-490e-b665-4712192e4d60","Type":"ContainerStarted","Data":"110e7b1ff7ca9f8d969f73bb89629c770d2aac04166a29e81ba2bd8dff0af2b0"}
Sep 15 18:59:31.386901 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:31.386839    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l" podStartSLOduration=1.6418297960000001 podStartE2EDuration="4.386818244s" totalImagesPullingTime="2.744988448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:59:31.384542608 +0000 UTC m=+533.661311464" watchObservedRunningTime="2026-09-15 18:59:31.386818244 +0000 UTC m=+533.663587101"
Sep 15 18:59:32.216801 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.216768    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:32.221308 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.221277    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:32.369659 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.369628    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:32.370466 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.370449    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 18:59:32.439361 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.439332    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5kvkk"]
Sep 15 18:59:32.442249 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.442224    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5kvkk"
Sep 15 18:59:32.444325 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.444285    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ncz6z\""
Sep 15 18:59:32.444456 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.444285    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 18:59:32.449982 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.449962    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5kvkk"]
Sep 15 18:59:32.517620 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.517486    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45964f19-df64-47fb-a693-ad00a923388a-config-file\") pod \"limitador-limitador-8588749977-5kvkk\" (UID: \"45964f19-df64-47fb-a693-ad00a923388a\") " pod="kuadrant-system/limitador-limitador-8588749977-5kvkk"
Sep 15 18:59:32.517620 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.517529    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp2qj\" (UniqueName: \"kubernetes.io/projected/45964f19-df64-47fb-a693-ad00a923388a-kube-api-access-tp2qj\") pod \"limitador-limitador-8588749977-5kvkk\" (UID: \"45964f19-df64-47fb-a693-ad00a923388a\") " pod="kuadrant-system/limitador-limitador-8588749977-5kvkk"
Sep 15 18:59:32.533571 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.533542    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5kvkk"]
Sep 15 18:59:32.618046 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.618017    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45964f19-df64-47fb-a693-ad00a923388a-config-file\") pod \"limitador-limitador-8588749977-5kvkk\" (UID: \"45964f19-df64-47fb-a693-ad00a923388a\") " pod="kuadrant-system/limitador-limitador-8588749977-5kvkk"
Sep 15 18:59:32.618046 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.618049    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp2qj\" (UniqueName: \"kubernetes.io/projected/45964f19-df64-47fb-a693-ad00a923388a-kube-api-access-tp2qj\") pod \"limitador-limitador-8588749977-5kvkk\" (UID: \"45964f19-df64-47fb-a693-ad00a923388a\") " pod="kuadrant-system/limitador-limitador-8588749977-5kvkk"
Sep 15 18:59:32.618629 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.618611    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45964f19-df64-47fb-a693-ad00a923388a-config-file\") pod \"limitador-limitador-8588749977-5kvkk\" (UID: \"45964f19-df64-47fb-a693-ad00a923388a\") " pod="kuadrant-system/limitador-limitador-8588749977-5kvkk"
Sep 15 18:59:32.625611 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.625585    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp2qj\" (UniqueName: \"kubernetes.io/projected/45964f19-df64-47fb-a693-ad00a923388a-kube-api-access-tp2qj\") pod \"limitador-limitador-8588749977-5kvkk\" (UID: \"45964f19-df64-47fb-a693-ad00a923388a\") " pod="kuadrant-system/limitador-limitador-8588749977-5kvkk"
Sep 15 18:59:32.753468 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.753423    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5kvkk"
Sep 15 18:59:32.836436 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.836407    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 18:59:32.840121 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.840102    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"
Sep 15 18:59:32.849027 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.849005    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 18:59:32.872910 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.872882    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 18:59:32.920607 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.920583    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cfe018cd-6e09-4ae5-9a76-1677382ff925-config-file\") pod \"limitador-limitador-84994f7ffc-sgjbh\" (UID: \"cfe018cd-6e09-4ae5-9a76-1677382ff925\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"
Sep 15 18:59:32.920714 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:32.920646    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htnch\" (UniqueName: \"kubernetes.io/projected/cfe018cd-6e09-4ae5-9a76-1677382ff925-kube-api-access-htnch\") pod \"limitador-limitador-84994f7ffc-sgjbh\" (UID: \"cfe018cd-6e09-4ae5-9a76-1677382ff925\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"
Sep 15 18:59:33.021157 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.021131    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htnch\" (UniqueName: \"kubernetes.io/projected/cfe018cd-6e09-4ae5-9a76-1677382ff925-kube-api-access-htnch\") pod \"limitador-limitador-84994f7ffc-sgjbh\" (UID: \"cfe018cd-6e09-4ae5-9a76-1677382ff925\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"
Sep 15 18:59:33.021273 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.021196    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cfe018cd-6e09-4ae5-9a76-1677382ff925-config-file\") pod \"limitador-limitador-84994f7ffc-sgjbh\" (UID: \"cfe018cd-6e09-4ae5-9a76-1677382ff925\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"
Sep 15 18:59:33.021752 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.021735    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cfe018cd-6e09-4ae5-9a76-1677382ff925-config-file\") pod \"limitador-limitador-84994f7ffc-sgjbh\" (UID: \"cfe018cd-6e09-4ae5-9a76-1677382ff925\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"
Sep 15 18:59:33.022035 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.022014    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5kvkk"]
Sep 15 18:59:33.024875 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:59:33.024847    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45964f19_df64_47fb_a693_ad00a923388a.slice/crio-3893a07ed11af2637afc936c1eef42b8d2f73cbcf8819d249e7f4f1fc30140c6 WatchSource:0}: Error finding container 3893a07ed11af2637afc936c1eef42b8d2f73cbcf8819d249e7f4f1fc30140c6: Status 404 returned error can't find the container with id 3893a07ed11af2637afc936c1eef42b8d2f73cbcf8819d249e7f4f1fc30140c6
Sep 15 18:59:33.028854 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.028775    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htnch\" (UniqueName: \"kubernetes.io/projected/cfe018cd-6e09-4ae5-9a76-1677382ff925-kube-api-access-htnch\") pod \"limitador-limitador-84994f7ffc-sgjbh\" (UID: \"cfe018cd-6e09-4ae5-9a76-1677382ff925\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"
Sep 15 18:59:33.151412 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.151356    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"
Sep 15 18:59:33.375109 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.375073    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5kvkk" event={"ID":"45964f19-df64-47fb-a693-ad00a923388a","Type":"ContainerStarted","Data":"3893a07ed11af2637afc936c1eef42b8d2f73cbcf8819d249e7f4f1fc30140c6"}
Sep 15 18:59:33.378844 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.378823    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-h9w5g"]
Sep 15 18:59:33.383138 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.383121    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-h9w5g"
Sep 15 18:59:33.385198 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.385179    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8r4b2\""
Sep 15 18:59:33.392723 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.390454    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-h9w5g"]
Sep 15 18:59:33.423413 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.423355    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 18:59:33.424596 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.424572    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mslc\" (UniqueName: \"kubernetes.io/projected/1353795f-cd5f-4e9c-9da1-811a0eedc410-kube-api-access-9mslc\") pod \"authorino-59c974bd48-h9w5g\" (UID: \"1353795f-cd5f-4e9c-9da1-811a0eedc410\") " pod="kuadrant-system/authorino-59c974bd48-h9w5g"
Sep 15 18:59:33.426184 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:59:33.426161    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe018cd_6e09_4ae5_9a76_1677382ff925.slice/crio-99d52285699f2238b486649d4d592b0d554ce2831966c1db2ba7c21953f14f0e WatchSource:0}: Error finding container 99d52285699f2238b486649d4d592b0d554ce2831966c1db2ba7c21953f14f0e: Status 404 returned error can't find the container with id 99d52285699f2238b486649d4d592b0d554ce2831966c1db2ba7c21953f14f0e
Sep 15 18:59:33.525837 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.525805    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mslc\" (UniqueName: \"kubernetes.io/projected/1353795f-cd5f-4e9c-9da1-811a0eedc410-kube-api-access-9mslc\") pod \"authorino-59c974bd48-h9w5g\" (UID: \"1353795f-cd5f-4e9c-9da1-811a0eedc410\") " pod="kuadrant-system/authorino-59c974bd48-h9w5g"
Sep 15 18:59:33.533079 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.533059    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mslc\" (UniqueName: \"kubernetes.io/projected/1353795f-cd5f-4e9c-9da1-811a0eedc410-kube-api-access-9mslc\") pod \"authorino-59c974bd48-h9w5g\" (UID: \"1353795f-cd5f-4e9c-9da1-811a0eedc410\") " pod="kuadrant-system/authorino-59c974bd48-h9w5g"
Sep 15 18:59:33.695423 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.695337    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-h9w5g"
Sep 15 18:59:33.963110 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:33.963023    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-h9w5g"]
Sep 15 18:59:33.966548 ip-10-0-129-227 kubenswrapper[2550]: W0915 18:59:33.966515    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1353795f_cd5f_4e9c_9da1_811a0eedc410.slice/crio-845256cde07fa4c3ed31bbea7b307a7fd045909a2d41734e638ad48b353ae0e1 WatchSource:0}: Error finding container 845256cde07fa4c3ed31bbea7b307a7fd045909a2d41734e638ad48b353ae0e1: Status 404 returned error can't find the container with id 845256cde07fa4c3ed31bbea7b307a7fd045909a2d41734e638ad48b353ae0e1
Sep 15 18:59:34.381219 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:34.381189    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5kvkk" event={"ID":"45964f19-df64-47fb-a693-ad00a923388a","Type":"ContainerStarted","Data":"9c1cc0203c9af0b31b213ca1bbac2329187f4f8a9fc486760be239f411d78f02"}
Sep 15 18:59:34.381612 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:34.381286    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-5kvkk"
Sep 15 18:59:34.382790 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:34.382767    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-sgjbh" event={"ID":"cfe018cd-6e09-4ae5-9a76-1677382ff925","Type":"ContainerStarted","Data":"c668de770fec587baa39233c44f02deacd633dfe1aa21201e2a45df1c09c1597"}
Sep 15 18:59:34.382892 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:34.382812    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"
Sep 15 18:59:34.382892 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:34.382833    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-sgjbh" event={"ID":"cfe018cd-6e09-4ae5-9a76-1677382ff925","Type":"ContainerStarted","Data":"99d52285699f2238b486649d4d592b0d554ce2831966c1db2ba7c21953f14f0e"}
Sep 15 18:59:34.384099 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:34.384076    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-h9w5g" event={"ID":"1353795f-cd5f-4e9c-9da1-811a0eedc410","Type":"ContainerStarted","Data":"845256cde07fa4c3ed31bbea7b307a7fd045909a2d41734e638ad48b353ae0e1"}
Sep 15 18:59:34.396633 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:34.396597    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-5kvkk" podStartSLOduration=1.18770666 podStartE2EDuration="2.396585525s" totalImagesPullingTime="1.208878865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:59:34.395337453 +0000 UTC m=+536.672106309" watchObservedRunningTime="2026-09-15 18:59:34.396585525 +0000 UTC m=+536.673354395"
Sep 15 18:59:34.410738 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:34.410700    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-sgjbh" podStartSLOduration=1.601760359 podStartE2EDuration="2.410687146s" totalImagesPullingTime="808.926787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:59:34.409113118 +0000 UTC m=+536.685881986" watchObservedRunningTime="2026-09-15 18:59:34.410687146 +0000 UTC m=+536.687456001"
Sep 15 18:59:36.396452 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:36.396413    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-h9w5g" event={"ID":"1353795f-cd5f-4e9c-9da1-811a0eedc410","Type":"ContainerStarted","Data":"9e3e3c81ee9e1f9578919417ddb07e15b5ef6e2c15b474d4b372e81f4a1b15e9"}
Sep 15 18:59:36.413709 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:36.413665    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-h9w5g" podStartSLOduration=1.7567102879999998 podStartE2EDuration="3.41365024s" totalImagesPullingTime="1.656939952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:59:36.41109378 +0000 UTC m=+538.687862636" watchObservedRunningTime="2026-09-15 18:59:36.41365024 +0000 UTC m=+538.690419096"
Sep 15 18:59:45.390339 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:45.390294    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-5kvkk"
Sep 15 18:59:45.390744 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:45.390381    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"
Sep 15 18:59:45.447448 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:45.447414    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5kvkk"]
Sep 15 18:59:45.447655 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:45.447622    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-5kvkk" podUID="45964f19-df64-47fb-a693-ad00a923388a" containerName="limitador" containerID="cri-o://9c1cc0203c9af0b31b213ca1bbac2329187f4f8a9fc486760be239f411d78f02" gracePeriod=30
Sep 15 18:59:45.989652 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:45.989632    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5kvkk"
Sep 15 18:59:46.128239 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:46.128204    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45964f19-df64-47fb-a693-ad00a923388a-config-file\") pod \"45964f19-df64-47fb-a693-ad00a923388a\" (UID: \"45964f19-df64-47fb-a693-ad00a923388a\") "
Sep 15 18:59:46.128453 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:46.128267    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp2qj\" (UniqueName: \"kubernetes.io/projected/45964f19-df64-47fb-a693-ad00a923388a-kube-api-access-tp2qj\") pod \"45964f19-df64-47fb-a693-ad00a923388a\" (UID: \"45964f19-df64-47fb-a693-ad00a923388a\") "
Sep 15 18:59:46.128576 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:46.128555    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45964f19-df64-47fb-a693-ad00a923388a-config-file" (OuterVolumeSpecName: "config-file") pod "45964f19-df64-47fb-a693-ad00a923388a" (UID: "45964f19-df64-47fb-a693-ad00a923388a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:59:46.130295 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:46.130274    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45964f19-df64-47fb-a693-ad00a923388a-kube-api-access-tp2qj" (OuterVolumeSpecName: "kube-api-access-tp2qj") pod "45964f19-df64-47fb-a693-ad00a923388a" (UID: "45964f19-df64-47fb-a693-ad00a923388a"). InnerVolumeSpecName "kube-api-access-tp2qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:59:46.229102 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:46.229074    2550 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45964f19-df64-47fb-a693-ad00a923388a-config-file\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:59:46.229102 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:46.229097    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp2qj\" (UniqueName: \"kubernetes.io/projected/45964f19-df64-47fb-a693-ad00a923388a-kube-api-access-tp2qj\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 18:59:46.439375 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:46.439275    2550 generic.go:356] "Generic (PLEG): container finished" podID="45964f19-df64-47fb-a693-ad00a923388a" containerID="9c1cc0203c9af0b31b213ca1bbac2329187f4f8a9fc486760be239f411d78f02" exitCode=0
Sep 15 18:59:46.439375 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:46.439327    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5kvkk"
Sep 15 18:59:46.439875 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:46.439322    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5kvkk" event={"ID":"45964f19-df64-47fb-a693-ad00a923388a","Type":"ContainerDied","Data":"9c1cc0203c9af0b31b213ca1bbac2329187f4f8a9fc486760be239f411d78f02"}
Sep 15 18:59:46.439875 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:46.439461    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5kvkk" event={"ID":"45964f19-df64-47fb-a693-ad00a923388a","Type":"ContainerDied","Data":"3893a07ed11af2637afc936c1eef42b8d2f73cbcf8819d249e7f4f1fc30140c6"}
Sep 15 18:59:46.439875 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:46.439498    2550 scope.go:117] "RemoveContainer" containerID="9c1cc0203c9af0b31b213ca1bbac2329187f4f8a9fc486760be239f411d78f02"
Sep 15 18:59:46.447398 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:46.447381    2550 scope.go:117] "RemoveContainer" containerID="9c1cc0203c9af0b31b213ca1bbac2329187f4f8a9fc486760be239f411d78f02"
Sep 15 18:59:46.447669 ip-10-0-129-227 kubenswrapper[2550]: E0915 18:59:46.447648    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1cc0203c9af0b31b213ca1bbac2329187f4f8a9fc486760be239f411d78f02\": container with ID starting with 9c1cc0203c9af0b31b213ca1bbac2329187f4f8a9fc486760be239f411d78f02 not found: ID does not exist" containerID="9c1cc0203c9af0b31b213ca1bbac2329187f4f8a9fc486760be239f411d78f02"
Sep 15 18:59:46.447753 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:46.447681    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1cc0203c9af0b31b213ca1bbac2329187f4f8a9fc486760be239f411d78f02"} err="failed to get container status \"9c1cc0203c9af0b31b213ca1bbac2329187f4f8a9fc486760be239f411d78f02\": rpc error: code = NotFound desc = could not find container \"9c1cc0203c9af0b31b213ca1bbac2329187f4f8a9fc486760be239f411d78f02\": container with ID starting with 9c1cc0203c9af0b31b213ca1bbac2329187f4f8a9fc486760be239f411d78f02 not found: ID does not exist"
Sep 15 18:59:46.454285 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:46.454265    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5kvkk"]
Sep 15 18:59:46.460226 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:46.460204    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5kvkk"]
Sep 15 18:59:48.294362 ip-10-0-129-227 kubenswrapper[2550]: I0915 18:59:48.294327    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45964f19-df64-47fb-a693-ad00a923388a" path="/var/lib/kubelet/pods/45964f19-df64-47fb-a693-ad00a923388a/volumes"
Sep 15 19:00:01.830849 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:01.830816    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-p8mwd"]
Sep 15 19:00:01.831284 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:01.831141    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45964f19-df64-47fb-a693-ad00a923388a" containerName="limitador"
Sep 15 19:00:01.831284 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:01.831153    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="45964f19-df64-47fb-a693-ad00a923388a" containerName="limitador"
Sep 15 19:00:01.831284 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:01.831208    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="45964f19-df64-47fb-a693-ad00a923388a" containerName="limitador"
Sep 15 19:00:01.835578 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:01.835562    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-p8mwd"
Sep 15 19:00:01.839543 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:01.839519    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-p8mwd"]
Sep 15 19:00:01.954102 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:01.954070    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-p8mwd"]
Sep 15 19:00:01.954346 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:00:01.954308    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-psldb], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-p8mwd" podUID="8dc8b507-3af3-4a51-abad-5968cb15db29"
Sep 15 19:00:01.955820 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:01.955793    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psldb\" (UniqueName: \"kubernetes.io/projected/8dc8b507-3af3-4a51-abad-5968cb15db29-kube-api-access-psldb\") pod \"authorino-557f876f69-p8mwd\" (UID: \"8dc8b507-3af3-4a51-abad-5968cb15db29\") " pod="kuadrant-system/authorino-557f876f69-p8mwd"
Sep 15 19:00:01.978830 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:01.978807    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2ppps"]
Sep 15 19:00:01.981970 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:01.981952    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2ppps"
Sep 15 19:00:01.984064 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:01.984041    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 19:00:01.989936 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:01.989912    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2ppps"]
Sep 15 19:00:02.056774 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.056743    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psldb\" (UniqueName: \"kubernetes.io/projected/8dc8b507-3af3-4a51-abad-5968cb15db29-kube-api-access-psldb\") pod \"authorino-557f876f69-p8mwd\" (UID: \"8dc8b507-3af3-4a51-abad-5968cb15db29\") " pod="kuadrant-system/authorino-557f876f69-p8mwd"
Sep 15 19:00:02.065141 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.065120    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psldb\" (UniqueName: \"kubernetes.io/projected/8dc8b507-3af3-4a51-abad-5968cb15db29-kube-api-access-psldb\") pod \"authorino-557f876f69-p8mwd\" (UID: \"8dc8b507-3af3-4a51-abad-5968cb15db29\") " pod="kuadrant-system/authorino-557f876f69-p8mwd"
Sep 15 19:00:02.095736 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.095680    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2ppps"]
Sep 15 19:00:02.095911 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:00:02.095893    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zhq8m tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-2ppps" podUID="08e9b796-942d-441f-a177-6cf0a727421f"
Sep 15 19:00:02.157691 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.157660    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhq8m\" (UniqueName: \"kubernetes.io/projected/08e9b796-942d-441f-a177-6cf0a727421f-kube-api-access-zhq8m\") pod \"authorino-6d4dfc649f-2ppps\" (UID: \"08e9b796-942d-441f-a177-6cf0a727421f\") " pod="kuadrant-system/authorino-6d4dfc649f-2ppps"
Sep 15 19:00:02.157797 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.157710    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08e9b796-942d-441f-a177-6cf0a727421f-tls-cert\") pod \"authorino-6d4dfc649f-2ppps\" (UID: \"08e9b796-942d-441f-a177-6cf0a727421f\") " pod="kuadrant-system/authorino-6d4dfc649f-2ppps"
Sep 15 19:00:02.259112 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.259082    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhq8m\" (UniqueName: \"kubernetes.io/projected/08e9b796-942d-441f-a177-6cf0a727421f-kube-api-access-zhq8m\") pod \"authorino-6d4dfc649f-2ppps\" (UID: \"08e9b796-942d-441f-a177-6cf0a727421f\") " pod="kuadrant-system/authorino-6d4dfc649f-2ppps"
Sep 15 19:00:02.259227 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.259130    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08e9b796-942d-441f-a177-6cf0a727421f-tls-cert\") pod \"authorino-6d4dfc649f-2ppps\" (UID: \"08e9b796-942d-441f-a177-6cf0a727421f\") " pod="kuadrant-system/authorino-6d4dfc649f-2ppps"
Sep 15 19:00:02.261471 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.261452    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08e9b796-942d-441f-a177-6cf0a727421f-tls-cert\") pod \"authorino-6d4dfc649f-2ppps\" (UID: \"08e9b796-942d-441f-a177-6cf0a727421f\") " pod="kuadrant-system/authorino-6d4dfc649f-2ppps"
Sep 15 19:00:02.269028 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.269004    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhq8m\" (UniqueName: \"kubernetes.io/projected/08e9b796-942d-441f-a177-6cf0a727421f-kube-api-access-zhq8m\") pod \"authorino-6d4dfc649f-2ppps\" (UID: \"08e9b796-942d-441f-a177-6cf0a727421f\") " pod="kuadrant-system/authorino-6d4dfc649f-2ppps"
Sep 15 19:00:02.505000 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.504926    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2ppps"
Sep 15 19:00:02.505150 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.504926    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-p8mwd"
Sep 15 19:00:02.526125 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.526099    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2ppps"
Sep 15 19:00:02.536684 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.536664    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-p8mwd"
Sep 15 19:00:02.662409 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.662379    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psldb\" (UniqueName: \"kubernetes.io/projected/8dc8b507-3af3-4a51-abad-5968cb15db29-kube-api-access-psldb\") pod \"8dc8b507-3af3-4a51-abad-5968cb15db29\" (UID: \"8dc8b507-3af3-4a51-abad-5968cb15db29\") "
Sep 15 19:00:02.662527 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.662438    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08e9b796-942d-441f-a177-6cf0a727421f-tls-cert\") pod \"08e9b796-942d-441f-a177-6cf0a727421f\" (UID: \"08e9b796-942d-441f-a177-6cf0a727421f\") "
Sep 15 19:00:02.662527 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.662499    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhq8m\" (UniqueName: \"kubernetes.io/projected/08e9b796-942d-441f-a177-6cf0a727421f-kube-api-access-zhq8m\") pod \"08e9b796-942d-441f-a177-6cf0a727421f\" (UID: \"08e9b796-942d-441f-a177-6cf0a727421f\") "
Sep 15 19:00:02.664486 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.664456    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e9b796-942d-441f-a177-6cf0a727421f-kube-api-access-zhq8m" (OuterVolumeSpecName: "kube-api-access-zhq8m") pod "08e9b796-942d-441f-a177-6cf0a727421f" (UID: "08e9b796-942d-441f-a177-6cf0a727421f"). InnerVolumeSpecName "kube-api-access-zhq8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:00:02.664486 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.664474    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08e9b796-942d-441f-a177-6cf0a727421f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "08e9b796-942d-441f-a177-6cf0a727421f" (UID: "08e9b796-942d-441f-a177-6cf0a727421f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:00:02.664609 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.664555    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc8b507-3af3-4a51-abad-5968cb15db29-kube-api-access-psldb" (OuterVolumeSpecName: "kube-api-access-psldb") pod "8dc8b507-3af3-4a51-abad-5968cb15db29" (UID: "8dc8b507-3af3-4a51-abad-5968cb15db29"). InnerVolumeSpecName "kube-api-access-psldb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:00:02.763975 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.763924    2550 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08e9b796-942d-441f-a177-6cf0a727421f-tls-cert\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:00:02.763975 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.763947    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhq8m\" (UniqueName: \"kubernetes.io/projected/08e9b796-942d-441f-a177-6cf0a727421f-kube-api-access-zhq8m\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:00:02.763975 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:02.763958    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psldb\" (UniqueName: \"kubernetes.io/projected/8dc8b507-3af3-4a51-abad-5968cb15db29-kube-api-access-psldb\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:00:03.508593 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:03.508564    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-p8mwd"
Sep 15 19:00:03.508989 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:03.508564    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2ppps"
Sep 15 19:00:03.540131 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:03.540101    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-p8mwd"]
Sep 15 19:00:03.544296 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:03.544272    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-p8mwd"]
Sep 15 19:00:03.562076 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:03.562054    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2ppps"]
Sep 15 19:00:03.564616 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:03.564593    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2ppps"]
Sep 15 19:00:03.657792 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:03.657760    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-h9w5g"]
Sep 15 19:00:03.657952 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:03.657929    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-h9w5g" podUID="1353795f-cd5f-4e9c-9da1-811a0eedc410" containerName="authorino" containerID="cri-o://9e3e3c81ee9e1f9578919417ddb07e15b5ef6e2c15b474d4b372e81f4a1b15e9" gracePeriod=30
Sep 15 19:00:03.898644 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:03.898623    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-h9w5g"
Sep 15 19:00:04.074701 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:04.074602    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mslc\" (UniqueName: \"kubernetes.io/projected/1353795f-cd5f-4e9c-9da1-811a0eedc410-kube-api-access-9mslc\") pod \"1353795f-cd5f-4e9c-9da1-811a0eedc410\" (UID: \"1353795f-cd5f-4e9c-9da1-811a0eedc410\") "
Sep 15 19:00:04.076651 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:04.076626    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1353795f-cd5f-4e9c-9da1-811a0eedc410-kube-api-access-9mslc" (OuterVolumeSpecName: "kube-api-access-9mslc") pod "1353795f-cd5f-4e9c-9da1-811a0eedc410" (UID: "1353795f-cd5f-4e9c-9da1-811a0eedc410"). InnerVolumeSpecName "kube-api-access-9mslc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:00:04.175153 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:04.175118    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mslc\" (UniqueName: \"kubernetes.io/projected/1353795f-cd5f-4e9c-9da1-811a0eedc410-kube-api-access-9mslc\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:00:04.294947 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:04.294916    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08e9b796-942d-441f-a177-6cf0a727421f" path="/var/lib/kubelet/pods/08e9b796-942d-441f-a177-6cf0a727421f/volumes"
Sep 15 19:00:04.295142 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:04.295130    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dc8b507-3af3-4a51-abad-5968cb15db29" path="/var/lib/kubelet/pods/8dc8b507-3af3-4a51-abad-5968cb15db29/volumes"
Sep 15 19:00:04.513840 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:04.513763    2550 generic.go:356] "Generic (PLEG): container finished" podID="1353795f-cd5f-4e9c-9da1-811a0eedc410" containerID="9e3e3c81ee9e1f9578919417ddb07e15b5ef6e2c15b474d4b372e81f4a1b15e9" exitCode=0
Sep 15 19:00:04.513840 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:04.513802    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-h9w5g"
Sep 15 19:00:04.513840 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:04.513810    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-h9w5g" event={"ID":"1353795f-cd5f-4e9c-9da1-811a0eedc410","Type":"ContainerDied","Data":"9e3e3c81ee9e1f9578919417ddb07e15b5ef6e2c15b474d4b372e81f4a1b15e9"}
Sep 15 19:00:04.513840 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:04.513841    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-h9w5g" event={"ID":"1353795f-cd5f-4e9c-9da1-811a0eedc410","Type":"ContainerDied","Data":"845256cde07fa4c3ed31bbea7b307a7fd045909a2d41734e638ad48b353ae0e1"}
Sep 15 19:00:04.514381 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:04.513857    2550 scope.go:117] "RemoveContainer" containerID="9e3e3c81ee9e1f9578919417ddb07e15b5ef6e2c15b474d4b372e81f4a1b15e9"
Sep 15 19:00:04.521131 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:04.521113    2550 scope.go:117] "RemoveContainer" containerID="9e3e3c81ee9e1f9578919417ddb07e15b5ef6e2c15b474d4b372e81f4a1b15e9"
Sep 15 19:00:04.521448 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:00:04.521429    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3e3c81ee9e1f9578919417ddb07e15b5ef6e2c15b474d4b372e81f4a1b15e9\": container with ID starting with 9e3e3c81ee9e1f9578919417ddb07e15b5ef6e2c15b474d4b372e81f4a1b15e9 not found: ID does not exist" containerID="9e3e3c81ee9e1f9578919417ddb07e15b5ef6e2c15b474d4b372e81f4a1b15e9"
Sep 15 19:00:04.521526 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:04.521457    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3e3c81ee9e1f9578919417ddb07e15b5ef6e2c15b474d4b372e81f4a1b15e9"} err="failed to get container status \"9e3e3c81ee9e1f9578919417ddb07e15b5ef6e2c15b474d4b372e81f4a1b15e9\": rpc error: code = NotFound desc = could not find container \"9e3e3c81ee9e1f9578919417ddb07e15b5ef6e2c15b474d4b372e81f4a1b15e9\": container with ID starting with 9e3e3c81ee9e1f9578919417ddb07e15b5ef6e2c15b474d4b372e81f4a1b15e9 not found: ID does not exist"
Sep 15 19:00:04.528305 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:04.528282    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-h9w5g"]
Sep 15 19:00:04.531704 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:04.531681    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-h9w5g"]
Sep 15 19:00:06.294264 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:06.294225    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1353795f-cd5f-4e9c-9da1-811a0eedc410" path="/var/lib/kubelet/pods/1353795f-cd5f-4e9c-9da1-811a0eedc410/volumes"
Sep 15 19:00:08.651408 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.651378    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-zrl7b"]
Sep 15 19:00:08.651910 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.651791    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1353795f-cd5f-4e9c-9da1-811a0eedc410" containerName="authorino"
Sep 15 19:00:08.651910 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.651811    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1353795f-cd5f-4e9c-9da1-811a0eedc410" containerName="authorino"
Sep 15 19:00:08.651910 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.651909    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1353795f-cd5f-4e9c-9da1-811a0eedc410" containerName="authorino"
Sep 15 19:00:08.657718 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.657697    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:08.659974 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.659948    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 19:00:08.659974 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.659969    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ftvtd\""
Sep 15 19:00:08.660144 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.660100    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 19:00:08.661052 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.661032    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 19:00:08.663121 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.663103    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-zrl7b"]
Sep 15 19:00:08.707958 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.707931    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6c7z\" (UniqueName: \"kubernetes.io/projected/fe21087f-b5f6-486a-8e65-f4715d0ed450-kube-api-access-z6c7z\") pod \"maas-controller-75dc6cbdf-zrl7b\" (UID: \"fe21087f-b5f6-486a-8e65-f4715d0ed450\") " pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:08.708071 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.707979    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-metrics-certs\") pod \"maas-controller-75dc6cbdf-zrl7b\" (UID: \"fe21087f-b5f6-486a-8e65-f4715d0ed450\") " pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:08.708148 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.708092    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-webhook-cert\") pod \"maas-controller-75dc6cbdf-zrl7b\" (UID: \"fe21087f-b5f6-486a-8e65-f4715d0ed450\") " pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:08.809191 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.809164    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6c7z\" (UniqueName: \"kubernetes.io/projected/fe21087f-b5f6-486a-8e65-f4715d0ed450-kube-api-access-z6c7z\") pod \"maas-controller-75dc6cbdf-zrl7b\" (UID: \"fe21087f-b5f6-486a-8e65-f4715d0ed450\") " pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:08.809292 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.809215    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-metrics-certs\") pod \"maas-controller-75dc6cbdf-zrl7b\" (UID: \"fe21087f-b5f6-486a-8e65-f4715d0ed450\") " pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:08.809292 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.809263    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-webhook-cert\") pod \"maas-controller-75dc6cbdf-zrl7b\" (UID: \"fe21087f-b5f6-486a-8e65-f4715d0ed450\") " pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:08.809450 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:00:08.809401    2550 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 15 19:00:08.809450 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:00:08.809403    2550 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 15 19:00:08.809522 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:00:08.809471    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-webhook-cert podName:fe21087f-b5f6-486a-8e65-f4715d0ed450 nodeName:}" failed. No retries permitted until 2026-09-15 19:00:09.309452446 +0000 UTC m=+571.586221283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-webhook-cert") pod "maas-controller-75dc6cbdf-zrl7b" (UID: "fe21087f-b5f6-486a-8e65-f4715d0ed450") : secret "maas-controller-webhook-cert" not found
Sep 15 19:00:08.809522 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:00:08.809488    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-metrics-certs podName:fe21087f-b5f6-486a-8e65-f4715d0ed450 nodeName:}" failed. No retries permitted until 2026-09-15 19:00:09.309479343 +0000 UTC m=+571.586248180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-metrics-certs") pod "maas-controller-75dc6cbdf-zrl7b" (UID: "fe21087f-b5f6-486a-8e65-f4715d0ed450") : secret "maas-controller-metrics-tls" not found
Sep 15 19:00:08.824768 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.824733    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6c7z\" (UniqueName: \"kubernetes.io/projected/fe21087f-b5f6-486a-8e65-f4715d0ed450-kube-api-access-z6c7z\") pod \"maas-controller-75dc6cbdf-zrl7b\" (UID: \"fe21087f-b5f6-486a-8e65-f4715d0ed450\") " pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:08.832140 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.832118    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6ff85d469c-6stt5"]
Sep 15 19:00:08.836548 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.836532    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff85d469c-6stt5"
Sep 15 19:00:08.841960 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.841930    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-6stt5"]
Sep 15 19:00:08.910193 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.910126    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-webhook-cert\") pod \"maas-controller-6ff85d469c-6stt5\" (UID: \"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b\") " pod="opendatahub/maas-controller-6ff85d469c-6stt5"
Sep 15 19:00:08.910193 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.910157    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-metrics-certs\") pod \"maas-controller-6ff85d469c-6stt5\" (UID: \"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b\") " pod="opendatahub/maas-controller-6ff85d469c-6stt5"
Sep 15 19:00:08.910193 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:08.910186    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmr84\" (UniqueName: \"kubernetes.io/projected/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-kube-api-access-nmr84\") pod \"maas-controller-6ff85d469c-6stt5\" (UID: \"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b\") " pod="opendatahub/maas-controller-6ff85d469c-6stt5"
Sep 15 19:00:09.010775 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:09.010743    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-webhook-cert\") pod \"maas-controller-6ff85d469c-6stt5\" (UID: \"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b\") " pod="opendatahub/maas-controller-6ff85d469c-6stt5"
Sep 15 19:00:09.010895 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:09.010789    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-metrics-certs\") pod \"maas-controller-6ff85d469c-6stt5\" (UID: \"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b\") " pod="opendatahub/maas-controller-6ff85d469c-6stt5"
Sep 15 19:00:09.010895 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:09.010833    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmr84\" (UniqueName: \"kubernetes.io/projected/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-kube-api-access-nmr84\") pod \"maas-controller-6ff85d469c-6stt5\" (UID: \"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b\") " pod="opendatahub/maas-controller-6ff85d469c-6stt5"
Sep 15 19:00:09.013280 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:09.013251    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-metrics-certs\") pod \"maas-controller-6ff85d469c-6stt5\" (UID: \"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b\") " pod="opendatahub/maas-controller-6ff85d469c-6stt5"
Sep 15 19:00:09.013399 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:09.013302    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-webhook-cert\") pod \"maas-controller-6ff85d469c-6stt5\" (UID: \"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b\") " pod="opendatahub/maas-controller-6ff85d469c-6stt5"
Sep 15 19:00:09.018249 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:09.018227    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmr84\" (UniqueName: \"kubernetes.io/projected/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-kube-api-access-nmr84\") pod \"maas-controller-6ff85d469c-6stt5\" (UID: \"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b\") " pod="opendatahub/maas-controller-6ff85d469c-6stt5"
Sep 15 19:00:09.149119 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:09.149087    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff85d469c-6stt5"
Sep 15 19:00:09.313506 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:09.313471    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-metrics-certs\") pod \"maas-controller-75dc6cbdf-zrl7b\" (UID: \"fe21087f-b5f6-486a-8e65-f4715d0ed450\") " pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:09.313656 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:09.313531    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-webhook-cert\") pod \"maas-controller-75dc6cbdf-zrl7b\" (UID: \"fe21087f-b5f6-486a-8e65-f4715d0ed450\") " pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:09.315905 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:09.315880    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-metrics-certs\") pod \"maas-controller-75dc6cbdf-zrl7b\" (UID: \"fe21087f-b5f6-486a-8e65-f4715d0ed450\") " pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:09.316008 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:09.315939    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-webhook-cert\") pod \"maas-controller-75dc6cbdf-zrl7b\" (UID: \"fe21087f-b5f6-486a-8e65-f4715d0ed450\") " pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:09.418597 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:09.418574    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-6stt5"]
Sep 15 19:00:09.420683 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:00:09.420661    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee7ef2c_f7ed_4dd3_b277_4e44e295a70b.slice/crio-50b02d9d534bcdffb683385b3cf9e4c4c333e6d54bed092fa07f8366bb4c3e03 WatchSource:0}: Error finding container 50b02d9d534bcdffb683385b3cf9e4c4c333e6d54bed092fa07f8366bb4c3e03: Status 404 returned error can't find the container with id 50b02d9d534bcdffb683385b3cf9e4c4c333e6d54bed092fa07f8366bb4c3e03
Sep 15 19:00:09.540237 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:09.540207    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff85d469c-6stt5" event={"ID":"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b","Type":"ContainerStarted","Data":"50b02d9d534bcdffb683385b3cf9e4c4c333e6d54bed092fa07f8366bb4c3e03"}
Sep 15 19:00:09.569169 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:09.569144    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:09.838095 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:09.838029    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-zrl7b"]
Sep 15 19:00:09.841721 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:00:09.841694    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe21087f_b5f6_486a_8e65_f4715d0ed450.slice/crio-03b40834d317eab548007cc4f8b307afbf8515e37e33f0084a23a245487a17f4 WatchSource:0}: Error finding container 03b40834d317eab548007cc4f8b307afbf8515e37e33f0084a23a245487a17f4: Status 404 returned error can't find the container with id 03b40834d317eab548007cc4f8b307afbf8515e37e33f0084a23a245487a17f4
Sep 15 19:00:10.545713 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:10.545673    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-zrl7b" event={"ID":"fe21087f-b5f6-486a-8e65-f4715d0ed450","Type":"ContainerStarted","Data":"03b40834d317eab548007cc4f8b307afbf8515e37e33f0084a23a245487a17f4"}
Sep 15 19:00:12.555979 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:12.555888    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff85d469c-6stt5" event={"ID":"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b","Type":"ContainerStarted","Data":"d5c9b50a5383afe95a2ac20db771ff28405af73d42b701ccc68f3e71873988c6"}
Sep 15 19:00:12.556434 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:12.556017    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6ff85d469c-6stt5"
Sep 15 19:00:12.557428 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:12.557393    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-zrl7b" event={"ID":"fe21087f-b5f6-486a-8e65-f4715d0ed450","Type":"ContainerStarted","Data":"2c8d23c55e623a65cd41518517353a7453c166b95b7bff7886aed755c1d32c8a"}
Sep 15 19:00:12.557558 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:12.557513    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:12.593111 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:12.593070    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6ff85d469c-6stt5" podStartSLOduration=1.725683093 podStartE2EDuration="4.593056709s" totalImagesPullingTime="2.867373616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:00:12.591020566 +0000 UTC m=+574.867789423" watchObservedRunningTime="2026-09-15 19:00:12.593056709 +0000 UTC m=+574.869825565"
Sep 15 19:00:12.625497 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:12.625451    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75dc6cbdf-zrl7b" podStartSLOduration=2.179609232 podStartE2EDuration="4.625437753s" totalImagesPullingTime="2.445828521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:00:12.623951516 +0000 UTC m=+574.900720371" watchObservedRunningTime="2026-09-15 19:00:12.625437753 +0000 UTC m=+574.902206611"
Sep 15 19:00:23.564358 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:23.564305    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6ff85d469c-6stt5"
Sep 15 19:00:23.564721 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:23.564556    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:23.628987 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:23.628953    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-zrl7b"]
Sep 15 19:00:23.629190 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:23.629166    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-75dc6cbdf-zrl7b" podUID="fe21087f-b5f6-486a-8e65-f4715d0ed450" containerName="manager" containerID="cri-o://2c8d23c55e623a65cd41518517353a7453c166b95b7bff7886aed755c1d32c8a" gracePeriod=30
Sep 15 19:00:23.870235 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:23.870211    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:23.928772 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:23.928740    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-webhook-cert\") pod \"fe21087f-b5f6-486a-8e65-f4715d0ed450\" (UID: \"fe21087f-b5f6-486a-8e65-f4715d0ed450\") "
Sep 15 19:00:23.928928 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:23.928803    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6c7z\" (UniqueName: \"kubernetes.io/projected/fe21087f-b5f6-486a-8e65-f4715d0ed450-kube-api-access-z6c7z\") pod \"fe21087f-b5f6-486a-8e65-f4715d0ed450\" (UID: \"fe21087f-b5f6-486a-8e65-f4715d0ed450\") "
Sep 15 19:00:23.928928 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:23.928882    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-metrics-certs\") pod \"fe21087f-b5f6-486a-8e65-f4715d0ed450\" (UID: \"fe21087f-b5f6-486a-8e65-f4715d0ed450\") "
Sep 15 19:00:23.930943 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:23.930910    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fe21087f-b5f6-486a-8e65-f4715d0ed450" (UID: "fe21087f-b5f6-486a-8e65-f4715d0ed450"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:00:23.931057 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:23.930961    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe21087f-b5f6-486a-8e65-f4715d0ed450-kube-api-access-z6c7z" (OuterVolumeSpecName: "kube-api-access-z6c7z") pod "fe21087f-b5f6-486a-8e65-f4715d0ed450" (UID: "fe21087f-b5f6-486a-8e65-f4715d0ed450"). InnerVolumeSpecName "kube-api-access-z6c7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:00:23.931057 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:23.930967    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fe21087f-b5f6-486a-8e65-f4715d0ed450" (UID: "fe21087f-b5f6-486a-8e65-f4715d0ed450"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:00:24.029625 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:24.029583    2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-metrics-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:00:24.029625 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:24.029618    2550 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe21087f-b5f6-486a-8e65-f4715d0ed450-webhook-cert\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:00:24.029625 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:24.029631    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6c7z\" (UniqueName: \"kubernetes.io/projected/fe21087f-b5f6-486a-8e65-f4715d0ed450-kube-api-access-z6c7z\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:00:24.608355 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:24.608267    2550 generic.go:356] "Generic (PLEG): container finished" podID="fe21087f-b5f6-486a-8e65-f4715d0ed450" containerID="2c8d23c55e623a65cd41518517353a7453c166b95b7bff7886aed755c1d32c8a" exitCode=0
Sep 15 19:00:24.608355 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:24.608328    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-zrl7b"
Sep 15 19:00:24.608767 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:24.608363    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-zrl7b" event={"ID":"fe21087f-b5f6-486a-8e65-f4715d0ed450","Type":"ContainerDied","Data":"2c8d23c55e623a65cd41518517353a7453c166b95b7bff7886aed755c1d32c8a"}
Sep 15 19:00:24.608767 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:24.608424    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-zrl7b" event={"ID":"fe21087f-b5f6-486a-8e65-f4715d0ed450","Type":"ContainerDied","Data":"03b40834d317eab548007cc4f8b307afbf8515e37e33f0084a23a245487a17f4"}
Sep 15 19:00:24.608767 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:24.608453    2550 scope.go:117] "RemoveContainer" containerID="2c8d23c55e623a65cd41518517353a7453c166b95b7bff7886aed755c1d32c8a"
Sep 15 19:00:24.616213 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:24.616189    2550 scope.go:117] "RemoveContainer" containerID="2c8d23c55e623a65cd41518517353a7453c166b95b7bff7886aed755c1d32c8a"
Sep 15 19:00:24.616493 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:00:24.616475    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8d23c55e623a65cd41518517353a7453c166b95b7bff7886aed755c1d32c8a\": container with ID starting with 2c8d23c55e623a65cd41518517353a7453c166b95b7bff7886aed755c1d32c8a not found: ID does not exist" containerID="2c8d23c55e623a65cd41518517353a7453c166b95b7bff7886aed755c1d32c8a"
Sep 15 19:00:24.616543 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:24.616502    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8d23c55e623a65cd41518517353a7453c166b95b7bff7886aed755c1d32c8a"} err="failed to get container status \"2c8d23c55e623a65cd41518517353a7453c166b95b7bff7886aed755c1d32c8a\": rpc error: code = NotFound desc = could not find container \"2c8d23c55e623a65cd41518517353a7453c166b95b7bff7886aed755c1d32c8a\": container with ID starting with 2c8d23c55e623a65cd41518517353a7453c166b95b7bff7886aed755c1d32c8a not found: ID does not exist"
Sep 15 19:00:24.625094 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:24.625075    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-zrl7b"]
Sep 15 19:00:24.629522 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:24.629503    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-zrl7b"]
Sep 15 19:00:26.294807 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:26.294773    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe21087f-b5f6-486a-8e65-f4715d0ed450" path="/var/lib/kubelet/pods/fe21087f-b5f6-486a-8e65-f4715d0ed450/volumes"
Sep 15 19:00:43.228798 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.228763    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"]
Sep 15 19:00:43.229233 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.229082    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe21087f-b5f6-486a-8e65-f4715d0ed450" containerName="manager"
Sep 15 19:00:43.229233 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.229093    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe21087f-b5f6-486a-8e65-f4715d0ed450" containerName="manager"
Sep 15 19:00:43.229233 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.229145    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe21087f-b5f6-486a-8e65-f4715d0ed450" containerName="manager"
Sep 15 19:00:43.233442 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.233416    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:43.236702 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.236680    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 19:00:43.236702 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.236692    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 19:00:43.236898 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.236735    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:00:43.236898 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.236681    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 19:00:43.236898 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.236681    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5ksbj\""
Sep 15 19:00:43.242774 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.242753    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"]
Sep 15 19:00:43.376002 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.375973    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afbf017-6c7f-45f6-9017-2ec92176ca0f-metrics-certs\") pod \"maas-api-59f6566bbc-f8b6f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:43.376155 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.376018    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0afbf017-6c7f-45f6-9017-2ec92176ca0f-maas-api-tls\") pod \"maas-api-59f6566bbc-f8b6f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:43.376155 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.376112    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxj2g\" (UniqueName: \"kubernetes.io/projected/0afbf017-6c7f-45f6-9017-2ec92176ca0f-kube-api-access-gxj2g\") pod \"maas-api-59f6566bbc-f8b6f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:43.376227 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.376152    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0afbf017-6c7f-45f6-9017-2ec92176ca0f-openshift-service-ca\") pod \"maas-api-59f6566bbc-f8b6f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:43.477049 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.477023    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afbf017-6c7f-45f6-9017-2ec92176ca0f-metrics-certs\") pod \"maas-api-59f6566bbc-f8b6f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:43.477210 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.477069    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0afbf017-6c7f-45f6-9017-2ec92176ca0f-maas-api-tls\") pod \"maas-api-59f6566bbc-f8b6f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:43.477210 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.477117    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxj2g\" (UniqueName: \"kubernetes.io/projected/0afbf017-6c7f-45f6-9017-2ec92176ca0f-kube-api-access-gxj2g\") pod \"maas-api-59f6566bbc-f8b6f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:43.477210 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.477150    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0afbf017-6c7f-45f6-9017-2ec92176ca0f-openshift-service-ca\") pod \"maas-api-59f6566bbc-f8b6f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:43.477853 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.477829    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0afbf017-6c7f-45f6-9017-2ec92176ca0f-openshift-service-ca\") pod \"maas-api-59f6566bbc-f8b6f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:43.479801 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.479740    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0afbf017-6c7f-45f6-9017-2ec92176ca0f-maas-api-tls\") pod \"maas-api-59f6566bbc-f8b6f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:43.479801 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.479772    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afbf017-6c7f-45f6-9017-2ec92176ca0f-metrics-certs\") pod \"maas-api-59f6566bbc-f8b6f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:43.485156 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.485131    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxj2g\" (UniqueName: \"kubernetes.io/projected/0afbf017-6c7f-45f6-9017-2ec92176ca0f-kube-api-access-gxj2g\") pod \"maas-api-59f6566bbc-f8b6f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:43.544787 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.544760    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:43.719102 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.719070    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:00:43.724450 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.724426    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-45sdg"]
Sep 15 19:00:43.728790 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.728772    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg"
Sep 15 19:00:43.730974 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.730927    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 15 19:00:43.731058 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.730979    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-zkwlt\""
Sep 15 19:00:43.739273 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.739253    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-45sdg"]
Sep 15 19:00:43.779351 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.779322    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rfkz\" (UniqueName: \"kubernetes.io/projected/c96e0591-bb2b-40ce-85c6-454e96e5489f-kube-api-access-7rfkz\") pod \"payload-processing-5b54b7fbb7-45sdg\" (UID: \"c96e0591-bb2b-40ce-85c6-454e96e5489f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg"
Sep 15 19:00:43.779486 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.779386    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96e0591-bb2b-40ce-85c6-454e96e5489f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-45sdg\" (UID: \"c96e0591-bb2b-40ce-85c6-454e96e5489f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg"
Sep 15 19:00:43.824710 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.824677    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"]
Sep 15 19:00:43.827745 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:00:43.827655    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0afbf017_6c7f_45f6_9017_2ec92176ca0f.slice/crio-4d71a01e2961890a48517ad6e7e8a581ed0ddbd0a95842ad45e05eda81e995bd WatchSource:0}: Error finding container 4d71a01e2961890a48517ad6e7e8a581ed0ddbd0a95842ad45e05eda81e995bd: Status 404 returned error can't find the container with id 4d71a01e2961890a48517ad6e7e8a581ed0ddbd0a95842ad45e05eda81e995bd
Sep 15 19:00:43.880322 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.880286    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rfkz\" (UniqueName: \"kubernetes.io/projected/c96e0591-bb2b-40ce-85c6-454e96e5489f-kube-api-access-7rfkz\") pod \"payload-processing-5b54b7fbb7-45sdg\" (UID: \"c96e0591-bb2b-40ce-85c6-454e96e5489f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg"
Sep 15 19:00:43.880446 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.880401    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96e0591-bb2b-40ce-85c6-454e96e5489f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-45sdg\" (UID: \"c96e0591-bb2b-40ce-85c6-454e96e5489f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg"
Sep 15 19:00:43.881040 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.881020    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96e0591-bb2b-40ce-85c6-454e96e5489f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-45sdg\" (UID: \"c96e0591-bb2b-40ce-85c6-454e96e5489f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg"
Sep 15 19:00:43.887993 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:43.887968    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rfkz\" (UniqueName: \"kubernetes.io/projected/c96e0591-bb2b-40ce-85c6-454e96e5489f-kube-api-access-7rfkz\") pod \"payload-processing-5b54b7fbb7-45sdg\" (UID: \"c96e0591-bb2b-40ce-85c6-454e96e5489f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg"
Sep 15 19:00:44.040557 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:44.040535    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg"
Sep 15 19:00:44.314050 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:44.314023    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-45sdg"]
Sep 15 19:00:44.316125 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:00:44.316098    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96e0591_bb2b_40ce_85c6_454e96e5489f.slice/crio-6224c01ca28cd815a746577637c308ed14dfc156c8caa29438a56d743fd5116e WatchSource:0}: Error finding container 6224c01ca28cd815a746577637c308ed14dfc156c8caa29438a56d743fd5116e: Status 404 returned error can't find the container with id 6224c01ca28cd815a746577637c308ed14dfc156c8caa29438a56d743fd5116e
Sep 15 19:00:44.710941 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:44.710795    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg" event={"ID":"c96e0591-bb2b-40ce-85c6-454e96e5489f","Type":"ContainerStarted","Data":"6224c01ca28cd815a746577637c308ed14dfc156c8caa29438a56d743fd5116e"}
Sep 15 19:00:44.713097 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:44.713056    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f" event={"ID":"0afbf017-6c7f-45f6-9017-2ec92176ca0f","Type":"ContainerStarted","Data":"4d71a01e2961890a48517ad6e7e8a581ed0ddbd0a95842ad45e05eda81e995bd"}
Sep 15 19:00:46.727718 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:46.727682    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f" event={"ID":"0afbf017-6c7f-45f6-9017-2ec92176ca0f","Type":"ContainerStarted","Data":"c6041f0426d70cdbd3e2276c3907f9a5c3cfb118125e915569b9cf50242d59cb"}
Sep 15 19:00:46.728188 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:46.727815    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:46.747749 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:46.747691    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f" podStartSLOduration=1.659761507 podStartE2EDuration="3.747675441s" totalImagesPullingTime="2.087913934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:00:46.745640086 +0000 UTC m=+609.022408942" watchObservedRunningTime="2026-09-15 19:00:46.747675441 +0000 UTC m=+609.024444295"
Sep 15 19:00:47.733718 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:47.733630    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg" event={"ID":"c96e0591-bb2b-40ce-85c6-454e96e5489f","Type":"ContainerStarted","Data":"fb6691a4c052459d45c17ca90b57323515244de90dc2fde84c325706d31d05ec"}
Sep 15 19:00:47.734135 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:47.733887    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg"
Sep 15 19:00:47.751066 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:47.751023    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg" podStartSLOduration=1.614415944 podStartE2EDuration="4.751008536s" totalImagesPullingTime="3.136592592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:00:47.749381433 +0000 UTC m=+610.026150290" watchObservedRunningTime="2026-09-15 19:00:47.751008536 +0000 UTC m=+610.027777395"
Sep 15 19:00:52.739262 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:52.739225    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:00:53.760470 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:53.760439    2550 generic.go:356] "Generic (PLEG): container finished" podID="8190ca1e-7e85-490e-b665-4712192e4d60" containerID="110e7b1ff7ca9f8d969f73bb89629c770d2aac04166a29e81ba2bd8dff0af2b0" exitCode=0
Sep 15 19:00:53.760982 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:53.760483    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l" event={"ID":"8190ca1e-7e85-490e-b665-4712192e4d60","Type":"ContainerDied","Data":"110e7b1ff7ca9f8d969f73bb89629c770d2aac04166a29e81ba2bd8dff0af2b0"}
Sep 15 19:00:53.760982 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:53.760924    2550 scope.go:117] "RemoveContainer" containerID="110e7b1ff7ca9f8d969f73bb89629c770d2aac04166a29e81ba2bd8dff0af2b0"
Sep 15 19:00:53.761929 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:53.761899    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:00:53.762006 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:53.761967    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:00:53.762006 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:53.762000    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:00:54.767481 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:54.767446    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l" event={"ID":"8190ca1e-7e85-490e-b665-4712192e4d60","Type":"ContainerStarted","Data":"2756d1ba43e03b3d837102376d9d1935f4e71ed1b657d2c2e70ed5ff88b3e452"}
Sep 15 19:00:55.216480 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:55.216443    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 19:00:55.216662 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:55.216644    2550 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-ht96l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 15 19:00:55.216706 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:55.216687    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l" podUID="8190ca1e-7e85-490e-b665-4712192e4d60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 15 19:00:56.216139 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:56.216104    2550 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-ht96l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 15 19:00:56.216543 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:56.216156    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l" podUID="8190ca1e-7e85-490e-b665-4712192e4d60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 15 19:00:57.216568 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:57.216521    2550 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-ht96l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 15 19:00:57.217034 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:57.216587    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l" podUID="8190ca1e-7e85-490e-b665-4712192e4d60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 15 19:00:58.216586 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:58.216542    2550 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-ht96l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 15 19:00:58.216965 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:58.216604    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l" podUID="8190ca1e-7e85-490e-b665-4712192e4d60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 15 19:00:58.740479 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:58.740444    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg"
Sep 15 19:00:59.216515 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:59.216472    2550 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-ht96l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 15 19:00:59.216696 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:59.216535    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l" podUID="8190ca1e-7e85-490e-b665-4712192e4d60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 15 19:00:59.718181 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:00:59.718147    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:01:00.220869 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:00.220839    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 19:01:00.221335 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:00.221209    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 19:01:00.221882 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:00.221863    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-ht96l"
Sep 15 19:01:09.804164 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:09.804127    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:01:25.407055 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:25.407017    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:01:31.836024 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.835984    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-976js"]
Sep 15 19:01:31.842965 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.842938    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.846083 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.846059    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 15 19:01:31.846225 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.846136    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 19:01:31.846419 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.846395    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 19:01:31.846535 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.846466    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w45l4\""
Sep 15 19:01:31.849265 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.849232    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-976js"]
Sep 15 19:01:31.883596 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.883561    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mspkd\" (UniqueName: \"kubernetes.io/projected/aae0751a-970a-4b4f-bb38-35d60332af2a-kube-api-access-mspkd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.883722 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.883609    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aae0751a-970a-4b4f-bb38-35d60332af2a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.883722 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.883638    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aae0751a-970a-4b4f-bb38-35d60332af2a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.883722 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.883684    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aae0751a-970a-4b4f-bb38-35d60332af2a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.883722 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.883716    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aae0751a-970a-4b4f-bb38-35d60332af2a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.883881 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.883801    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aae0751a-970a-4b4f-bb38-35d60332af2a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.984200 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.984173    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aae0751a-970a-4b4f-bb38-35d60332af2a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.984347 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.984217    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mspkd\" (UniqueName: \"kubernetes.io/projected/aae0751a-970a-4b4f-bb38-35d60332af2a-kube-api-access-mspkd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.984347 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.984235    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aae0751a-970a-4b4f-bb38-35d60332af2a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.984347 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.984271    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aae0751a-970a-4b4f-bb38-35d60332af2a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.984347 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.984302    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aae0751a-970a-4b4f-bb38-35d60332af2a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.984611 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.984349    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aae0751a-970a-4b4f-bb38-35d60332af2a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.984685 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.984660    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aae0751a-970a-4b4f-bb38-35d60332af2a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.984746 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.984711    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aae0751a-970a-4b4f-bb38-35d60332af2a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.984799 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.984752    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aae0751a-970a-4b4f-bb38-35d60332af2a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.986475 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.986451    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aae0751a-970a-4b4f-bb38-35d60332af2a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.986951 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.986929    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aae0751a-970a-4b4f-bb38-35d60332af2a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:31.992130 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:31.992110    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mspkd\" (UniqueName: \"kubernetes.io/projected/aae0751a-970a-4b4f-bb38-35d60332af2a-kube-api-access-mspkd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-976js\" (UID: \"aae0751a-970a-4b4f-bb38-35d60332af2a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:32.155127 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:32.155056    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:32.424659 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:32.424636    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-976js"]
Sep 15 19:01:32.427026 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:01:32.426998    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae0751a_970a_4b4f_bb38_35d60332af2a.slice/crio-cbc55b339fd49fff4b868a4a7e4d8d6af14cd021ec84dd8dce93fe9bcfc2a9c7 WatchSource:0}: Error finding container cbc55b339fd49fff4b868a4a7e4d8d6af14cd021ec84dd8dce93fe9bcfc2a9c7: Status 404 returned error can't find the container with id cbc55b339fd49fff4b868a4a7e4d8d6af14cd021ec84dd8dce93fe9bcfc2a9c7
Sep 15 19:01:32.930025 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:32.929985    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js" event={"ID":"aae0751a-970a-4b4f-bb38-35d60332af2a","Type":"ContainerStarted","Data":"cbc55b339fd49fff4b868a4a7e4d8d6af14cd021ec84dd8dce93fe9bcfc2a9c7"}
Sep 15 19:01:33.016953 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:33.016917    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:01:33.936721 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:33.936685    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js" event={"ID":"aae0751a-970a-4b4f-bb38-35d60332af2a","Type":"ContainerStarted","Data":"bf4409ee864e37183fcb4b784634745bc2f24a458dff702a20bae84d96b9bd83"}
Sep 15 19:01:33.956307 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:33.956261    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js" podStartSLOduration=1.9859542829999999 podStartE2EDuration="2.956245754s" totalImagesPullingTime="970.291471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:01:33.955242165 +0000 UTC m=+656.232011023" watchObservedRunningTime="2026-09-15 19:01:33.956245754 +0000 UTC m=+656.233014608"
Sep 15 19:01:42.155594 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:42.155545    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:42.155973 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:42.155613    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:42.167748 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:42.167726    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:42.986430 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:42.986389    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-976js"
Sep 15 19:01:43.832079 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:43.832044    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:01:53.024133 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.024099    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"]
Sep 15 19:01:53.029265 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.029240    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.031508 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.031488    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 15 19:01:53.039038 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.039013    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"]
Sep 15 19:01:53.165157 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.165120    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4530c92f-a596-49e0-bd79-fd69012c4cf3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.165157 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.165162    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4530c92f-a596-49e0-bd79-fd69012c4cf3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.165388 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.165226    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4530c92f-a596-49e0-bd79-fd69012c4cf3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.165388 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.165265    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4530c92f-a596-49e0-bd79-fd69012c4cf3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.165388 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.165292    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4530c92f-a596-49e0-bd79-fd69012c4cf3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.165490 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.165420    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jlvw\" (UniqueName: \"kubernetes.io/projected/4530c92f-a596-49e0-bd79-fd69012c4cf3-kube-api-access-2jlvw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.265910 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.265880    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jlvw\" (UniqueName: \"kubernetes.io/projected/4530c92f-a596-49e0-bd79-fd69012c4cf3-kube-api-access-2jlvw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.266052 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.265921    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4530c92f-a596-49e0-bd79-fd69012c4cf3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.266052 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.265938    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4530c92f-a596-49e0-bd79-fd69012c4cf3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.266052 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.265983    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4530c92f-a596-49e0-bd79-fd69012c4cf3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.266052 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.266029    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4530c92f-a596-49e0-bd79-fd69012c4cf3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.266204 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.266055    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4530c92f-a596-49e0-bd79-fd69012c4cf3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.266362 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.266339    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4530c92f-a596-49e0-bd79-fd69012c4cf3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.266488 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.266457    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4530c92f-a596-49e0-bd79-fd69012c4cf3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.266612 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.266465    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4530c92f-a596-49e0-bd79-fd69012c4cf3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.268239 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.268213    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4530c92f-a596-49e0-bd79-fd69012c4cf3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.268516 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.268498    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4530c92f-a596-49e0-bd79-fd69012c4cf3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.274372 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.274304    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jlvw\" (UniqueName: \"kubernetes.io/projected/4530c92f-a596-49e0-bd79-fd69012c4cf3-kube-api-access-2jlvw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g9qh8\" (UID: \"4530c92f-a596-49e0-bd79-fd69012c4cf3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.340951 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.340917    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:01:53.609646 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.609615    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"]
Sep 15 19:01:53.611068 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:01:53.611042    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4530c92f_a596_49e0_bd79_fd69012c4cf3.slice/crio-558a85ba80003723463fcf4e5f715b333ec5ea27ca709494df070ccf4635ab19 WatchSource:0}: Error finding container 558a85ba80003723463fcf4e5f715b333ec5ea27ca709494df070ccf4635ab19: Status 404 returned error can't find the container with id 558a85ba80003723463fcf4e5f715b333ec5ea27ca709494df070ccf4635ab19
Sep 15 19:01:53.612508 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:53.612489    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:01:54.022417 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:54.022387    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8" event={"ID":"4530c92f-a596-49e0-bd79-fd69012c4cf3","Type":"ContainerStarted","Data":"524c6e3cdb4bcf96e58b1ab638cc3086ee9166c34fb1aacc6213ed9e7800fd9b"}
Sep 15 19:01:54.022561 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:54.022447    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8" event={"ID":"4530c92f-a596-49e0-bd79-fd69012c4cf3","Type":"ContainerStarted","Data":"558a85ba80003723463fcf4e5f715b333ec5ea27ca709494df070ccf4635ab19"}
Sep 15 19:01:54.040297 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:54.040249    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8" podStartSLOduration=0.860583926 podStartE2EDuration="1.040234399s" totalImagesPullingTime="179.650473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:01:54.039693049 +0000 UTC m=+676.316461905" watchObservedRunningTime="2026-09-15 19:01:54.040234399 +0000 UTC m=+676.317003295"
Sep 15 19:01:54.710220 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:54.710184    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:01:56.239835 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.239801    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"]
Sep 15 19:01:56.243070 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.243054    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.245558 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.245532    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 15 19:01:56.252265 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.252244    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"]
Sep 15 19:01:56.395853 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.395819    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0629218b-cefa-4ba4-a166-87103c9c9952-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.395853 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.395856    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0629218b-cefa-4ba4-a166-87103c9c9952-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.396098 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.395896    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0629218b-cefa-4ba4-a166-87103c9c9952-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.396098 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.396022    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bl2v\" (UniqueName: \"kubernetes.io/projected/0629218b-cefa-4ba4-a166-87103c9c9952-kube-api-access-9bl2v\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.396098 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.396069    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0629218b-cefa-4ba4-a166-87103c9c9952-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.396218 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.396116    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0629218b-cefa-4ba4-a166-87103c9c9952-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.497279 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.497204    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0629218b-cefa-4ba4-a166-87103c9c9952-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.497279 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.497268    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bl2v\" (UniqueName: \"kubernetes.io/projected/0629218b-cefa-4ba4-a166-87103c9c9952-kube-api-access-9bl2v\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.497521 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.497288    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0629218b-cefa-4ba4-a166-87103c9c9952-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.497521 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.497342    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0629218b-cefa-4ba4-a166-87103c9c9952-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.497521 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.497386    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0629218b-cefa-4ba4-a166-87103c9c9952-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.497521 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.497408    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0629218b-cefa-4ba4-a166-87103c9c9952-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.497736 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.497711    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0629218b-cefa-4ba4-a166-87103c9c9952-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.497793 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.497775    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0629218b-cefa-4ba4-a166-87103c9c9952-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.497852 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.497775    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0629218b-cefa-4ba4-a166-87103c9c9952-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.499525 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.499504    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0629218b-cefa-4ba4-a166-87103c9c9952-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.499903 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.499880    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0629218b-cefa-4ba4-a166-87103c9c9952-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.506881 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.506859    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bl2v\" (UniqueName: \"kubernetes.io/projected/0629218b-cefa-4ba4-a166-87103c9c9952-kube-api-access-9bl2v\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-prwbx\" (UID: \"0629218b-cefa-4ba4-a166-87103c9c9952\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.552466 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.552431    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:01:56.821722 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:56.821697    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"]
Sep 15 19:01:56.823672 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:01:56.823643    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0629218b_cefa_4ba4_a166_87103c9c9952.slice/crio-1f35cf0c7ca77c4365f038ca038a178a416315f9fb881fce2b5fb79336cd5185 WatchSource:0}: Error finding container 1f35cf0c7ca77c4365f038ca038a178a416315f9fb881fce2b5fb79336cd5185: Status 404 returned error can't find the container with id 1f35cf0c7ca77c4365f038ca038a178a416315f9fb881fce2b5fb79336cd5185
Sep 15 19:01:57.037801 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:57.037775    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx" event={"ID":"0629218b-cefa-4ba4-a166-87103c9c9952","Type":"ContainerStarted","Data":"1f35cf0c7ca77c4365f038ca038a178a416315f9fb881fce2b5fb79336cd5185"}
Sep 15 19:01:58.046225 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:58.046189    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx" event={"ID":"0629218b-cefa-4ba4-a166-87103c9c9952","Type":"ContainerStarted","Data":"c496bde29d82503599e8e9f101f386ba7f6fb7f1643617dab2d3da436b7f3e0e"}
Sep 15 19:01:58.064408 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:01:58.064360    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx" podStartSLOduration=1.892219243 podStartE2EDuration="2.064345995s" totalImagesPullingTime="172.126752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:01:58.06310333 +0000 UTC m=+680.339872185" watchObservedRunningTime="2026-09-15 19:01:58.064345995 +0000 UTC m=+680.341114870"
Sep 15 19:02:02.411117 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:02:02.411079    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:02:03.341379 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:02:03.341345    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:02:03.341555 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:02:03.341397    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:02:03.353678 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:02:03.353651    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:02:04.083248 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:02:04.083214    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g9qh8"
Sep 15 19:02:06.552832 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:02:06.552796    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:02:06.553217 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:02:06.552851    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:02:06.565124 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:02:06.565101    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:02:07.102266 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:02:07.102233    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-prwbx"
Sep 15 19:03:08.625914 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:08.625829    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55c58d9c44-ddcsp"]
Sep 15 19:03:08.629168 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:08.629147    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55c58d9c44-ddcsp"
Sep 15 19:03:08.631199 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:08.631177    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 19:03:08.631971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:08.631948    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8r4b2\""
Sep 15 19:03:08.636829 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:08.636808    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55c58d9c44-ddcsp"]
Sep 15 19:03:08.721144 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:08.721116    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8cb3dc4-59a9-4174-ae0e-bb38a3088dbc-tls-cert\") pod \"authorino-55c58d9c44-ddcsp\" (UID: \"e8cb3dc4-59a9-4174-ae0e-bb38a3088dbc\") " pod="kuadrant-system/authorino-55c58d9c44-ddcsp"
Sep 15 19:03:08.721289 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:08.721170    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k27r\" (UniqueName: \"kubernetes.io/projected/e8cb3dc4-59a9-4174-ae0e-bb38a3088dbc-kube-api-access-6k27r\") pod \"authorino-55c58d9c44-ddcsp\" (UID: \"e8cb3dc4-59a9-4174-ae0e-bb38a3088dbc\") " pod="kuadrant-system/authorino-55c58d9c44-ddcsp"
Sep 15 19:03:08.822015 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:08.821981    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8cb3dc4-59a9-4174-ae0e-bb38a3088dbc-tls-cert\") pod \"authorino-55c58d9c44-ddcsp\" (UID: \"e8cb3dc4-59a9-4174-ae0e-bb38a3088dbc\") " pod="kuadrant-system/authorino-55c58d9c44-ddcsp"
Sep 15 19:03:08.822181 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:08.822046    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k27r\" (UniqueName: \"kubernetes.io/projected/e8cb3dc4-59a9-4174-ae0e-bb38a3088dbc-kube-api-access-6k27r\") pod \"authorino-55c58d9c44-ddcsp\" (UID: \"e8cb3dc4-59a9-4174-ae0e-bb38a3088dbc\") " pod="kuadrant-system/authorino-55c58d9c44-ddcsp"
Sep 15 19:03:08.824572 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:08.824546    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8cb3dc4-59a9-4174-ae0e-bb38a3088dbc-tls-cert\") pod \"authorino-55c58d9c44-ddcsp\" (UID: \"e8cb3dc4-59a9-4174-ae0e-bb38a3088dbc\") " pod="kuadrant-system/authorino-55c58d9c44-ddcsp"
Sep 15 19:03:08.829423 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:08.829396    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k27r\" (UniqueName: \"kubernetes.io/projected/e8cb3dc4-59a9-4174-ae0e-bb38a3088dbc-kube-api-access-6k27r\") pod \"authorino-55c58d9c44-ddcsp\" (UID: \"e8cb3dc4-59a9-4174-ae0e-bb38a3088dbc\") " pod="kuadrant-system/authorino-55c58d9c44-ddcsp"
Sep 15 19:03:08.938737 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:08.938658    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55c58d9c44-ddcsp"
Sep 15 19:03:09.206074 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:09.206048    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55c58d9c44-ddcsp"]
Sep 15 19:03:09.208566 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:03:09.208541    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8cb3dc4_59a9_4174_ae0e_bb38a3088dbc.slice/crio-0f0888b2450d2e51f547c2e5b9d9b63ad14ce0f02ea8854415fcbb3b0262d77a WatchSource:0}: Error finding container 0f0888b2450d2e51f547c2e5b9d9b63ad14ce0f02ea8854415fcbb3b0262d77a: Status 404 returned error can't find the container with id 0f0888b2450d2e51f547c2e5b9d9b63ad14ce0f02ea8854415fcbb3b0262d77a
Sep 15 19:03:09.373829 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:09.373797    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55c58d9c44-ddcsp" event={"ID":"e8cb3dc4-59a9-4174-ae0e-bb38a3088dbc","Type":"ContainerStarted","Data":"0f0888b2450d2e51f547c2e5b9d9b63ad14ce0f02ea8854415fcbb3b0262d77a"}
Sep 15 19:03:11.386682 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:11.386641    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55c58d9c44-ddcsp" event={"ID":"e8cb3dc4-59a9-4174-ae0e-bb38a3088dbc","Type":"ContainerStarted","Data":"9ab2526f282301655e95f6a0c4683032687f9003be78cc40c09eacafa5200594"}
Sep 15 19:03:11.404175 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:11.404120    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55c58d9c44-ddcsp" podStartSLOduration=1.9787063630000001 podStartE2EDuration="3.404105291s" totalImagesPullingTime="1.425398928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:03:11.402404943 +0000 UTC m=+753.679173800" watchObservedRunningTime="2026-09-15 19:03:11.404105291 +0000 UTC m=+753.680874153"
Sep 15 19:03:11.940338 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:11.940281    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-45sdg"]
Sep 15 19:03:11.940830 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:11.940775    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg" podUID="c96e0591-bb2b-40ce-85c6-454e96e5489f" containerName="payload-processing" containerID="cri-o://fb6691a4c052459d45c17ca90b57323515244de90dc2fde84c325706d31d05ec" gracePeriod=30
Sep 15 19:03:12.004411 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.004347    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-6stt5"]
Sep 15 19:03:12.006506 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.006386    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6ff85d469c-6stt5" podUID="5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b" containerName="manager" containerID="cri-o://d5c9b50a5383afe95a2ac20db771ff28405af73d42b701ccc68f3e71873988c6" gracePeriod=30
Sep 15 19:03:12.260874 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.260848    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg"
Sep 15 19:03:12.296378 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.296349    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff85d469c-6stt5"
Sep 15 19:03:12.353774 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.353738    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rfkz\" (UniqueName: \"kubernetes.io/projected/c96e0591-bb2b-40ce-85c6-454e96e5489f-kube-api-access-7rfkz\") pod \"c96e0591-bb2b-40ce-85c6-454e96e5489f\" (UID: \"c96e0591-bb2b-40ce-85c6-454e96e5489f\") "
Sep 15 19:03:12.353949 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.353786    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96e0591-bb2b-40ce-85c6-454e96e5489f-plugins-config-volume\") pod \"c96e0591-bb2b-40ce-85c6-454e96e5489f\" (UID: \"c96e0591-bb2b-40ce-85c6-454e96e5489f\") "
Sep 15 19:03:12.353949 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.353823    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmr84\" (UniqueName: \"kubernetes.io/projected/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-kube-api-access-nmr84\") pod \"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b\" (UID: \"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b\") "
Sep 15 19:03:12.353949 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.353883    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-webhook-cert\") pod \"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b\" (UID: \"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b\") "
Sep 15 19:03:12.353949 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.353908    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-metrics-certs\") pod \"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b\" (UID: \"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b\") "
Sep 15 19:03:12.354511 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.354159    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c96e0591-bb2b-40ce-85c6-454e96e5489f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c96e0591-bb2b-40ce-85c6-454e96e5489f" (UID: "c96e0591-bb2b-40ce-85c6-454e96e5489f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:03:12.354677 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.354651    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96e0591-bb2b-40ce-85c6-454e96e5489f-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:03:12.356517 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.356484    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c96e0591-bb2b-40ce-85c6-454e96e5489f-kube-api-access-7rfkz" (OuterVolumeSpecName: "kube-api-access-7rfkz") pod "c96e0591-bb2b-40ce-85c6-454e96e5489f" (UID: "c96e0591-bb2b-40ce-85c6-454e96e5489f"). InnerVolumeSpecName "kube-api-access-7rfkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:03:12.356678 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.356652    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b" (UID: "5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:03:12.356743 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.356719    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-kube-api-access-nmr84" (OuterVolumeSpecName: "kube-api-access-nmr84") pod "5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b" (UID: "5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b"). InnerVolumeSpecName "kube-api-access-nmr84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:03:12.356970 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.356948    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b" (UID: "5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:03:12.392178 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.392145    2550 generic.go:356] "Generic (PLEG): container finished" podID="c96e0591-bb2b-40ce-85c6-454e96e5489f" containerID="fb6691a4c052459d45c17ca90b57323515244de90dc2fde84c325706d31d05ec" exitCode=0
Sep 15 19:03:12.392602 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.392194    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg"
Sep 15 19:03:12.392602 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.392237    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg" event={"ID":"c96e0591-bb2b-40ce-85c6-454e96e5489f","Type":"ContainerDied","Data":"fb6691a4c052459d45c17ca90b57323515244de90dc2fde84c325706d31d05ec"}
Sep 15 19:03:12.392602 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.392298    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-45sdg" event={"ID":"c96e0591-bb2b-40ce-85c6-454e96e5489f","Type":"ContainerDied","Data":"6224c01ca28cd815a746577637c308ed14dfc156c8caa29438a56d743fd5116e"}
Sep 15 19:03:12.392602 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.392338    2550 scope.go:117] "RemoveContainer" containerID="fb6691a4c052459d45c17ca90b57323515244de90dc2fde84c325706d31d05ec"
Sep 15 19:03:12.393669 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.393641    2550 generic.go:356] "Generic (PLEG): container finished" podID="5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b" containerID="d5c9b50a5383afe95a2ac20db771ff28405af73d42b701ccc68f3e71873988c6" exitCode=0
Sep 15 19:03:12.393778 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.393693    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff85d469c-6stt5"
Sep 15 19:03:12.393778 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.393720    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff85d469c-6stt5" event={"ID":"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b","Type":"ContainerDied","Data":"d5c9b50a5383afe95a2ac20db771ff28405af73d42b701ccc68f3e71873988c6"}
Sep 15 19:03:12.393881 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.393781    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff85d469c-6stt5" event={"ID":"5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b","Type":"ContainerDied","Data":"50b02d9d534bcdffb683385b3cf9e4c4c333e6d54bed092fa07f8366bb4c3e03"}
Sep 15 19:03:12.402162 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.402143    2550 scope.go:117] "RemoveContainer" containerID="fb6691a4c052459d45c17ca90b57323515244de90dc2fde84c325706d31d05ec"
Sep 15 19:03:12.402479 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:03:12.402459    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6691a4c052459d45c17ca90b57323515244de90dc2fde84c325706d31d05ec\": container with ID starting with fb6691a4c052459d45c17ca90b57323515244de90dc2fde84c325706d31d05ec not found: ID does not exist" containerID="fb6691a4c052459d45c17ca90b57323515244de90dc2fde84c325706d31d05ec"
Sep 15 19:03:12.402572 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.402489    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6691a4c052459d45c17ca90b57323515244de90dc2fde84c325706d31d05ec"} err="failed to get container status \"fb6691a4c052459d45c17ca90b57323515244de90dc2fde84c325706d31d05ec\": rpc error: code = NotFound desc = could not find container \"fb6691a4c052459d45c17ca90b57323515244de90dc2fde84c325706d31d05ec\": container with ID starting with fb6691a4c052459d45c17ca90b57323515244de90dc2fde84c325706d31d05ec not found: ID does not exist"
Sep 15 19:03:12.402572 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.402511    2550 scope.go:117] "RemoveContainer" containerID="d5c9b50a5383afe95a2ac20db771ff28405af73d42b701ccc68f3e71873988c6"
Sep 15 19:03:12.412191 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.412163    2550 scope.go:117] "RemoveContainer" containerID="d5c9b50a5383afe95a2ac20db771ff28405af73d42b701ccc68f3e71873988c6"
Sep 15 19:03:12.412472 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:03:12.412453    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c9b50a5383afe95a2ac20db771ff28405af73d42b701ccc68f3e71873988c6\": container with ID starting with d5c9b50a5383afe95a2ac20db771ff28405af73d42b701ccc68f3e71873988c6 not found: ID does not exist" containerID="d5c9b50a5383afe95a2ac20db771ff28405af73d42b701ccc68f3e71873988c6"
Sep 15 19:03:12.412549 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.412478    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c9b50a5383afe95a2ac20db771ff28405af73d42b701ccc68f3e71873988c6"} err="failed to get container status \"d5c9b50a5383afe95a2ac20db771ff28405af73d42b701ccc68f3e71873988c6\": rpc error: code = NotFound desc = could not find container \"d5c9b50a5383afe95a2ac20db771ff28405af73d42b701ccc68f3e71873988c6\": container with ID starting with d5c9b50a5383afe95a2ac20db771ff28405af73d42b701ccc68f3e71873988c6 not found: ID does not exist"
Sep 15 19:03:12.413754 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.413732    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-45sdg"]
Sep 15 19:03:12.417221 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.417198    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-45sdg"]
Sep 15 19:03:12.427779 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.427736    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-6stt5"]
Sep 15 19:03:12.429895 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.429873    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-6stt5"]
Sep 15 19:03:12.455476 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.455443    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rfkz\" (UniqueName: \"kubernetes.io/projected/c96e0591-bb2b-40ce-85c6-454e96e5489f-kube-api-access-7rfkz\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:03:12.455476 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.455478    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmr84\" (UniqueName: \"kubernetes.io/projected/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-kube-api-access-nmr84\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:03:12.455663 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.455495    2550 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-webhook-cert\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:03:12.455663 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:12.455511    2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b-metrics-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:03:13.172215 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.172183    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6ff85d469c-tkww4"]
Sep 15 19:03:13.172554 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.172540    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b" containerName="manager"
Sep 15 19:03:13.172602 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.172555    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b" containerName="manager"
Sep 15 19:03:13.172602 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.172563    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c96e0591-bb2b-40ce-85c6-454e96e5489f" containerName="payload-processing"
Sep 15 19:03:13.172602 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.172568    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c96e0591-bb2b-40ce-85c6-454e96e5489f" containerName="payload-processing"
Sep 15 19:03:13.172694 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.172621    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b" containerName="manager"
Sep 15 19:03:13.172694 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.172630    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c96e0591-bb2b-40ce-85c6-454e96e5489f" containerName="payload-processing"
Sep 15 19:03:13.177013 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.176996    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff85d469c-tkww4"
Sep 15 19:03:13.180007 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.179989    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 19:03:13.180337 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.180306    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ftvtd\""
Sep 15 19:03:13.180438 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.180340    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 19:03:13.180438 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.180353    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 19:03:13.185657 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.185635    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-tkww4"]
Sep 15 19:03:13.263456 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.263370    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2666f189-73a4-4051-b0f2-1552f138aec8-webhook-cert\") pod \"maas-controller-6ff85d469c-tkww4\" (UID: \"2666f189-73a4-4051-b0f2-1552f138aec8\") " pod="opendatahub/maas-controller-6ff85d469c-tkww4"
Sep 15 19:03:13.263456 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.263406    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2666f189-73a4-4051-b0f2-1552f138aec8-metrics-certs\") pod \"maas-controller-6ff85d469c-tkww4\" (UID: \"2666f189-73a4-4051-b0f2-1552f138aec8\") " pod="opendatahub/maas-controller-6ff85d469c-tkww4"
Sep 15 19:03:13.263456 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.263434    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkdkw\" (UniqueName: \"kubernetes.io/projected/2666f189-73a4-4051-b0f2-1552f138aec8-kube-api-access-xkdkw\") pod \"maas-controller-6ff85d469c-tkww4\" (UID: \"2666f189-73a4-4051-b0f2-1552f138aec8\") " pod="opendatahub/maas-controller-6ff85d469c-tkww4"
Sep 15 19:03:13.364540 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.364504    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2666f189-73a4-4051-b0f2-1552f138aec8-webhook-cert\") pod \"maas-controller-6ff85d469c-tkww4\" (UID: \"2666f189-73a4-4051-b0f2-1552f138aec8\") " pod="opendatahub/maas-controller-6ff85d469c-tkww4"
Sep 15 19:03:13.364540 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.364543    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2666f189-73a4-4051-b0f2-1552f138aec8-metrics-certs\") pod \"maas-controller-6ff85d469c-tkww4\" (UID: \"2666f189-73a4-4051-b0f2-1552f138aec8\") " pod="opendatahub/maas-controller-6ff85d469c-tkww4"
Sep 15 19:03:13.364770 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.364645    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkdkw\" (UniqueName: \"kubernetes.io/projected/2666f189-73a4-4051-b0f2-1552f138aec8-kube-api-access-xkdkw\") pod \"maas-controller-6ff85d469c-tkww4\" (UID: \"2666f189-73a4-4051-b0f2-1552f138aec8\") " pod="opendatahub/maas-controller-6ff85d469c-tkww4"
Sep 15 19:03:13.367028 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.366999    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2666f189-73a4-4051-b0f2-1552f138aec8-webhook-cert\") pod \"maas-controller-6ff85d469c-tkww4\" (UID: \"2666f189-73a4-4051-b0f2-1552f138aec8\") " pod="opendatahub/maas-controller-6ff85d469c-tkww4"
Sep 15 19:03:13.367127 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.367060    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2666f189-73a4-4051-b0f2-1552f138aec8-metrics-certs\") pod \"maas-controller-6ff85d469c-tkww4\" (UID: \"2666f189-73a4-4051-b0f2-1552f138aec8\") " pod="opendatahub/maas-controller-6ff85d469c-tkww4"
Sep 15 19:03:13.374425 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.374405    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkdkw\" (UniqueName: \"kubernetes.io/projected/2666f189-73a4-4051-b0f2-1552f138aec8-kube-api-access-xkdkw\") pod \"maas-controller-6ff85d469c-tkww4\" (UID: \"2666f189-73a4-4051-b0f2-1552f138aec8\") " pod="opendatahub/maas-controller-6ff85d469c-tkww4"
Sep 15 19:03:13.487320 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.487294    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff85d469c-tkww4"
Sep 15 19:03:13.753749 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:13.753726    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-tkww4"]
Sep 15 19:03:13.755497 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:03:13.755467    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2666f189_73a4_4051_b0f2_1552f138aec8.slice/crio-0d5e584043611d738268198d1617309c267072272021d690d4a39732bc1bdc2a WatchSource:0}: Error finding container 0d5e584043611d738268198d1617309c267072272021d690d4a39732bc1bdc2a: Status 404 returned error can't find the container with id 0d5e584043611d738268198d1617309c267072272021d690d4a39732bc1bdc2a
Sep 15 19:03:14.295821 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:14.295793    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b" path="/var/lib/kubelet/pods/5ee7ef2c-f7ed-4dd3-b277-4e44e295a70b/volumes"
Sep 15 19:03:14.296132 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:14.296113    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c96e0591-bb2b-40ce-85c6-454e96e5489f" path="/var/lib/kubelet/pods/c96e0591-bb2b-40ce-85c6-454e96e5489f/volumes"
Sep 15 19:03:14.405777 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:14.405741    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff85d469c-tkww4" event={"ID":"2666f189-73a4-4051-b0f2-1552f138aec8","Type":"ContainerStarted","Data":"e52c970ba07ed68ca1cbeb7b54e22b21fa48c8352a59751f41b772507288cbde"}
Sep 15 19:03:14.405941 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:14.405789    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff85d469c-tkww4" event={"ID":"2666f189-73a4-4051-b0f2-1552f138aec8","Type":"ContainerStarted","Data":"0d5e584043611d738268198d1617309c267072272021d690d4a39732bc1bdc2a"}
Sep 15 19:03:14.405941 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:14.405874    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6ff85d469c-tkww4"
Sep 15 19:03:14.424288 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:14.424245    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6ff85d469c-tkww4" podStartSLOduration=0.968392588 podStartE2EDuration="1.424232082s" totalImagesPullingTime="455.839494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:03:14.422129024 +0000 UTC m=+756.698897879" watchObservedRunningTime="2026-09-15 19:03:14.424232082 +0000 UTC m=+756.701000984"
Sep 15 19:03:25.412726 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:03:25.412697    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6ff85d469c-tkww4"
Sep 15 19:04:14.352988 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.352902    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"]
Sep 15 19:04:14.356774 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.356751    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.359153 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.359133    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-74ef1f-openshift-default-dockercfg-nc5ph\""
Sep 15 19:04:14.367035 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.367014    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"]
Sep 15 19:04:14.481164 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.481125    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-workload-socket\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.481369 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.481201    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-workload-certs\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.481369 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.481234    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flvgc\" (UniqueName: \"kubernetes.io/projected/ec73c8b1-223d-4859-8a85-d2faeabd20c9-kube-api-access-flvgc\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.481369 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.481267    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-envoy\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.481369 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.481338    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-credential-socket\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.481662 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.481385    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-token\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.481662 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.481432    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-data\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.481662 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.481455    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-podinfo\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.481662 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.481484    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istiod-ca-cert\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.582502 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.582469    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-data\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.582690 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.582508    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-podinfo\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.582690 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.582538    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istiod-ca-cert\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.582690 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.582589    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-workload-socket\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.582851 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.582750    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-workload-certs\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.582851 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.582811    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flvgc\" (UniqueName: \"kubernetes.io/projected/ec73c8b1-223d-4859-8a85-d2faeabd20c9-kube-api-access-flvgc\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.582958 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.582907    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-data\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.583015 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.582979    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-workload-socket\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.583127 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.583097    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-workload-certs\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.583222 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.582846    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-envoy\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.583284 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.583209    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-credential-socket\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.583603 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.583559    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-credential-socket\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.584016 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.583959    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istiod-ca-cert\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.584146 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.584119    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-token\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.585582 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.585558    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-envoy\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.586719 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.586694    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-podinfo\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.592858 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.592834    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-token\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.593010 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.592880    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flvgc\" (UniqueName: \"kubernetes.io/projected/ec73c8b1-223d-4859-8a85-d2faeabd20c9-kube-api-access-flvgc\") pod \"e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") " pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.668377 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.668323    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:14.941444 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.941414    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"]
Sep 15 19:04:14.943470 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:04:14.943438    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec73c8b1_223d_4859_8a85_d2faeabd20c9.slice/crio-98df13e04a129551f5229b1a0c23a3497efe0f82da46316f40bf840e6c04c5a3 WatchSource:0}: Error finding container 98df13e04a129551f5229b1a0c23a3497efe0f82da46316f40bf840e6c04c5a3: Status 404 returned error can't find the container with id 98df13e04a129551f5229b1a0c23a3497efe0f82da46316f40bf840e6c04c5a3
Sep 15 19:04:14.945729 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.945688    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:04:14.945798 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.945764    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:04:14.945798 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:14.945791    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:04:15.682237 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:15.682197    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99" event={"ID":"ec73c8b1-223d-4859-8a85-d2faeabd20c9","Type":"ContainerStarted","Data":"7aacf60a3dee30353f3fe29939cfdfbb253b997a13abe797c9d2dceaef9a9ec4"}
Sep 15 19:04:15.682705 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:15.682264    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99" event={"ID":"ec73c8b1-223d-4859-8a85-d2faeabd20c9","Type":"ContainerStarted","Data":"98df13e04a129551f5229b1a0c23a3497efe0f82da46316f40bf840e6c04c5a3"}
Sep 15 19:04:15.716858 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:15.716799    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99" podStartSLOduration=1.716779293 podStartE2EDuration="1.716779293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:15.714499484 +0000 UTC m=+817.991268354" watchObservedRunningTime="2026-09-15 19:04:15.716779293 +0000 UTC m=+817.993548150"
Sep 15 19:04:16.669551 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:16.669512    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:16.674538 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:16.674513    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:16.686547 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:16.686519    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:16.687457 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:16.687436    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:19.511463 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.511426    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"]
Sep 15 19:04:19.514903 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.514885    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"
Sep 15 19:04:19.517049 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.517029    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-7c96ca-dockercfg-94sm6\""
Sep 15 19:04:19.517134 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.517068    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-7c96ca\""
Sep 15 19:04:19.523788 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.523765    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"]
Sep 15 19:04:19.629575 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.629541    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637a8186-5537-4288-9b91-44610151ec81-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl\" (UID: \"637a8186-5537-4288-9b91-44610151ec81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"
Sep 15 19:04:19.629794 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.629639    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlrxw\" (UniqueName: \"kubernetes.io/projected/637a8186-5537-4288-9b91-44610151ec81-kube-api-access-nlrxw\") pod \"payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl\" (UID: \"637a8186-5537-4288-9b91-44610151ec81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"
Sep 15 19:04:19.730515 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.730483    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrxw\" (UniqueName: \"kubernetes.io/projected/637a8186-5537-4288-9b91-44610151ec81-kube-api-access-nlrxw\") pod \"payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl\" (UID: \"637a8186-5537-4288-9b91-44610151ec81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"
Sep 15 19:04:19.730681 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.730582    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637a8186-5537-4288-9b91-44610151ec81-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl\" (UID: \"637a8186-5537-4288-9b91-44610151ec81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"
Sep 15 19:04:19.731245 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.731220    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637a8186-5537-4288-9b91-44610151ec81-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl\" (UID: \"637a8186-5537-4288-9b91-44610151ec81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"
Sep 15 19:04:19.741166 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.741142    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlrxw\" (UniqueName: \"kubernetes.io/projected/637a8186-5537-4288-9b91-44610151ec81-kube-api-access-nlrxw\") pod \"payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl\" (UID: \"637a8186-5537-4288-9b91-44610151ec81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"
Sep 15 19:04:19.744779 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.744752    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"]
Sep 15 19:04:19.748526 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.748506    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:19.750642 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.750620    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-74ef1f-serving-cert\""
Sep 15 19:04:19.759820 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.759737    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"]
Sep 15 19:04:19.827558 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.827534    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"
Sep 15 19:04:19.831757 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.831738    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc77959-6e75-41db-8767-858a6f6c9052-maas-api-tls\") pod \"maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:19.831840 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.831766    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc77959-6e75-41db-8767-858a6f6c9052-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:19.831840 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.831783    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mfww\" (UniqueName: \"kubernetes.io/projected/0cc77959-6e75-41db-8767-858a6f6c9052-kube-api-access-7mfww\") pod \"maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:19.831957 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.831929    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cc77959-6e75-41db-8767-858a6f6c9052-metrics-certs\") pod \"maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:19.932658 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.932621    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cc77959-6e75-41db-8767-858a6f6c9052-metrics-certs\") pod \"maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:19.932807 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.932697    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc77959-6e75-41db-8767-858a6f6c9052-maas-api-tls\") pod \"maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:19.932807 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.932724    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc77959-6e75-41db-8767-858a6f6c9052-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:19.932807 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.932743    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mfww\" (UniqueName: \"kubernetes.io/projected/0cc77959-6e75-41db-8767-858a6f6c9052-kube-api-access-7mfww\") pod \"maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:19.933014 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:04:19.932853    2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-serving-cert: secret "maas-api-e2e-worker-w0-74ef1f-serving-cert" not found
Sep 15 19:04:19.933014 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:04:19.932932    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cc77959-6e75-41db-8767-858a6f6c9052-maas-api-tls podName:0cc77959-6e75-41db-8767-858a6f6c9052 nodeName:}" failed. No retries permitted until 2026-09-15 19:04:20.432910237 +0000 UTC m=+822.709679076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0cc77959-6e75-41db-8767-858a6f6c9052-maas-api-tls") pod "maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv" (UID: "0cc77959-6e75-41db-8767-858a6f6c9052") : secret "maas-api-e2e-worker-w0-74ef1f-serving-cert" not found
Sep 15 19:04:19.933502 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.933475    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc77959-6e75-41db-8767-858a6f6c9052-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:19.935186 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.935170    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cc77959-6e75-41db-8767-858a6f6c9052-metrics-certs\") pod \"maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:19.941119 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:19.941096    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mfww\" (UniqueName: \"kubernetes.io/projected/0cc77959-6e75-41db-8767-858a6f6c9052-kube-api-access-7mfww\") pod \"maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:20.101494 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.101458    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"]
Sep 15 19:04:20.103499 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:04:20.103476    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637a8186_5537_4288_9b91_44610151ec81.slice/crio-ab3df66aa026b3d41b12966b25b941e8c21d24612aa203d4030daf0e52b10e1d WatchSource:0}: Error finding container ab3df66aa026b3d41b12966b25b941e8c21d24612aa203d4030daf0e52b10e1d: Status 404 returned error can't find the container with id ab3df66aa026b3d41b12966b25b941e8c21d24612aa203d4030daf0e52b10e1d
Sep 15 19:04:20.437148 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.437071    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc77959-6e75-41db-8767-858a6f6c9052-maas-api-tls\") pod \"maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:20.439499 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.439466    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc77959-6e75-41db-8767-858a6f6c9052-maas-api-tls\") pod \"maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:20.440431 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.440408    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"]
Sep 15 19:04:20.443851 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.443833    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"
Sep 15 19:04:20.446376 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.446352    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-74ef1f\""
Sep 15 19:04:20.446499 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.446457    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-74ef1f-dockercfg-b7gl8\""
Sep 15 19:04:20.458006 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.457982    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"]
Sep 15 19:04:20.537705 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.537664    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e739f55c-0ca9-442e-b6d0-dc8144acafd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp\" (UID: \"e739f55c-0ca9-442e-b6d0-dc8144acafd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"
Sep 15 19:04:20.537705 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.537707    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psq79\" (UniqueName: \"kubernetes.io/projected/e739f55c-0ca9-442e-b6d0-dc8144acafd0-kube-api-access-psq79\") pod \"payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp\" (UID: \"e739f55c-0ca9-442e-b6d0-dc8144acafd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"
Sep 15 19:04:20.638934 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.638894    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e739f55c-0ca9-442e-b6d0-dc8144acafd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp\" (UID: \"e739f55c-0ca9-442e-b6d0-dc8144acafd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"
Sep 15 19:04:20.639103 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.638944    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psq79\" (UniqueName: \"kubernetes.io/projected/e739f55c-0ca9-442e-b6d0-dc8144acafd0-kube-api-access-psq79\") pod \"payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp\" (UID: \"e739f55c-0ca9-442e-b6d0-dc8144acafd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"
Sep 15 19:04:20.639657 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.639630    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e739f55c-0ca9-442e-b6d0-dc8144acafd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp\" (UID: \"e739f55c-0ca9-442e-b6d0-dc8144acafd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"
Sep 15 19:04:20.647002 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.646973    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psq79\" (UniqueName: \"kubernetes.io/projected/e739f55c-0ca9-442e-b6d0-dc8144acafd0-kube-api-access-psq79\") pod \"payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp\" (UID: \"e739f55c-0ca9-442e-b6d0-dc8144acafd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"
Sep 15 19:04:20.666547 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.666525    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:20.707172 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.707101    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl" event={"ID":"637a8186-5537-4288-9b91-44610151ec81","Type":"ContainerStarted","Data":"6438624ff3045fe266ddb1abcbfd2832bcf46cb95e05dc8a6a3280b1937b7dcc"}
Sep 15 19:04:20.707172 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.707151    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl" event={"ID":"637a8186-5537-4288-9b91-44610151ec81","Type":"ContainerStarted","Data":"ab3df66aa026b3d41b12966b25b941e8c21d24612aa203d4030daf0e52b10e1d"}
Sep 15 19:04:20.707370 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.707350    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"
Sep 15 19:04:20.726411 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.725479    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl" podStartSLOduration=1.72545927 podStartE2EDuration="1.72545927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:20.720711342 +0000 UTC m=+822.997480198" watchObservedRunningTime="2026-09-15 19:04:20.72545927 +0000 UTC m=+823.002228128"
Sep 15 19:04:20.753938 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.753914    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"
Sep 15 19:04:20.945423 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:20.945346    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"]
Sep 15 19:04:20.947729 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:04:20.947704    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc77959_6e75_41db_8767_858a6f6c9052.slice/crio-4ad152980ff32693afec7f6b207f52b8c0164160d2ee276eba9f6e6f7d4d9f2f WatchSource:0}: Error finding container 4ad152980ff32693afec7f6b207f52b8c0164160d2ee276eba9f6e6f7d4d9f2f: Status 404 returned error can't find the container with id 4ad152980ff32693afec7f6b207f52b8c0164160d2ee276eba9f6e6f7d4d9f2f
Sep 15 19:04:21.041890 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.041867    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"]
Sep 15 19:04:21.043990 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:04:21.043964    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode739f55c_0ca9_442e_b6d0_dc8144acafd0.slice/crio-c8d2bb754d8d49e0ebbdbd6221cd55112b9ba22290ca691bb306cb0fdd2a2dab WatchSource:0}: Error finding container c8d2bb754d8d49e0ebbdbd6221cd55112b9ba22290ca691bb306cb0fdd2a2dab: Status 404 returned error can't find the container with id c8d2bb754d8d49e0ebbdbd6221cd55112b9ba22290ca691bb306cb0fdd2a2dab
Sep 15 19:04:21.382067 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.380897    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"]
Sep 15 19:04:21.385811 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.385789    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"
Sep 15 19:04:21.388087 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.388063    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d5559a\""
Sep 15 19:04:21.388197 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.388116    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d5559a-dockercfg-hvl25\""
Sep 15 19:04:21.394356 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.394334    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"]
Sep 15 19:04:21.413333 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.413272    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"]
Sep 15 19:04:21.417068 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.417045    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"
Sep 15 19:04:21.426568 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.426542    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"]
Sep 15 19:04:21.547871 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.547832    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfxnz\" (UniqueName: \"kubernetes.io/projected/59bdbca8-25a1-486c-acdd-b2b2db5ab220-kube-api-access-pfxnz\") pod \"payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp\" (UID: \"59bdbca8-25a1-486c-acdd-b2b2db5ab220\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"
Sep 15 19:04:21.548287 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.547876    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz88b\" (UniqueName: \"kubernetes.io/projected/7ba7584f-beec-478c-b6f2-e2976ca9487a-kube-api-access-rz88b\") pod \"payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7\" (UID: \"7ba7584f-beec-478c-b6f2-e2976ca9487a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"
Sep 15 19:04:21.548287 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.548057    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59bdbca8-25a1-486c-acdd-b2b2db5ab220-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp\" (UID: \"59bdbca8-25a1-486c-acdd-b2b2db5ab220\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"
Sep 15 19:04:21.548287 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.548105    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba7584f-beec-478c-b6f2-e2976ca9487a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7\" (UID: \"7ba7584f-beec-478c-b6f2-e2976ca9487a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"
Sep 15 19:04:21.649197 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.649127    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz88b\" (UniqueName: \"kubernetes.io/projected/7ba7584f-beec-478c-b6f2-e2976ca9487a-kube-api-access-rz88b\") pod \"payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7\" (UID: \"7ba7584f-beec-478c-b6f2-e2976ca9487a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"
Sep 15 19:04:21.649367 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.649239    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59bdbca8-25a1-486c-acdd-b2b2db5ab220-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp\" (UID: \"59bdbca8-25a1-486c-acdd-b2b2db5ab220\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"
Sep 15 19:04:21.649367 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.649278    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba7584f-beec-478c-b6f2-e2976ca9487a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7\" (UID: \"7ba7584f-beec-478c-b6f2-e2976ca9487a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"
Sep 15 19:04:21.649465 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.649363    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfxnz\" (UniqueName: \"kubernetes.io/projected/59bdbca8-25a1-486c-acdd-b2b2db5ab220-kube-api-access-pfxnz\") pod \"payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp\" (UID: \"59bdbca8-25a1-486c-acdd-b2b2db5ab220\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"
Sep 15 19:04:21.650208 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.650184    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59bdbca8-25a1-486c-acdd-b2b2db5ab220-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp\" (UID: \"59bdbca8-25a1-486c-acdd-b2b2db5ab220\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"
Sep 15 19:04:21.650534 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.650184    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba7584f-beec-478c-b6f2-e2976ca9487a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7\" (UID: \"7ba7584f-beec-478c-b6f2-e2976ca9487a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"
Sep 15 19:04:21.659735 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.659711    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz88b\" (UniqueName: \"kubernetes.io/projected/7ba7584f-beec-478c-b6f2-e2976ca9487a-kube-api-access-rz88b\") pod \"payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7\" (UID: \"7ba7584f-beec-478c-b6f2-e2976ca9487a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"
Sep 15 19:04:21.659831 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.659769    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfxnz\" (UniqueName: \"kubernetes.io/projected/59bdbca8-25a1-486c-acdd-b2b2db5ab220-kube-api-access-pfxnz\") pod \"payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp\" (UID: \"59bdbca8-25a1-486c-acdd-b2b2db5ab220\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"
Sep 15 19:04:21.714963 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.714939    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"
Sep 15 19:04:21.715983 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.715957    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv" event={"ID":"0cc77959-6e75-41db-8767-858a6f6c9052","Type":"ContainerStarted","Data":"5d61df8d4fe609e48a0da4ff5f4e7b87f66a8402694d0e5264bd7373534c3cd0"}
Sep 15 19:04:21.716108 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.716008    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv" event={"ID":"0cc77959-6e75-41db-8767-858a6f6c9052","Type":"ContainerStarted","Data":"4ad152980ff32693afec7f6b207f52b8c0164160d2ee276eba9f6e6f7d4d9f2f"}
Sep 15 19:04:21.716180 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.716113    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:21.717696 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.717670    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp" event={"ID":"e739f55c-0ca9-442e-b6d0-dc8144acafd0","Type":"ContainerStarted","Data":"a9db53c8dd3afe9bbded6d21a35afb2d6dd727659a2559777451eaf238b5e717"}
Sep 15 19:04:21.717796 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.717719    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp" event={"ID":"e739f55c-0ca9-442e-b6d0-dc8144acafd0","Type":"ContainerStarted","Data":"c8d2bb754d8d49e0ebbdbd6221cd55112b9ba22290ca691bb306cb0fdd2a2dab"}
Sep 15 19:04:21.717861 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.717828    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"
Sep 15 19:04:21.729950 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.729840    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"
Sep 15 19:04:21.733030 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.732984    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv" podStartSLOduration=2.316951775 podStartE2EDuration="2.732966713s" totalImagesPullingTime="416.014938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:21.729896646 +0000 UTC m=+824.006665503" watchObservedRunningTime="2026-09-15 19:04:21.732966713 +0000 UTC m=+824.009735570"
Sep 15 19:04:21.990601 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.990557    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp" podStartSLOduration=1.990538403 podStartE2EDuration="1.990538403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:21.746851649 +0000 UTC m=+824.023620506" watchObservedRunningTime="2026-09-15 19:04:21.990538403 +0000 UTC m=+824.267307253"
Sep 15 19:04:21.990808 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:21.990795    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"]
Sep 15 19:04:21.992660 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:04:21.992627    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59bdbca8_25a1_486c_acdd_b2b2db5ab220.slice/crio-43c82f034fdfa70509bbc8dd9ed65cf436e82830a1b53095273ed4a7db03c144 WatchSource:0}: Error finding container 43c82f034fdfa70509bbc8dd9ed65cf436e82830a1b53095273ed4a7db03c144: Status 404 returned error can't find the container with id 43c82f034fdfa70509bbc8dd9ed65cf436e82830a1b53095273ed4a7db03c144
Sep 15 19:04:22.012636 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:22.012608    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"]
Sep 15 19:04:22.015777 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:04:22.015753    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba7584f_beec_478c_b6f2_e2976ca9487a.slice/crio-c5cbf16860591e0c984de81eb1bf8f0c476112e1127021e79b17defd37a1d659 WatchSource:0}: Error finding container c5cbf16860591e0c984de81eb1bf8f0c476112e1127021e79b17defd37a1d659: Status 404 returned error can't find the container with id c5cbf16860591e0c984de81eb1bf8f0c476112e1127021e79b17defd37a1d659
Sep 15 19:04:22.724568 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:22.724534    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7" event={"ID":"7ba7584f-beec-478c-b6f2-e2976ca9487a","Type":"ContainerStarted","Data":"82eb0db1b9b384cd4c4978b4850b55e2b690e27fdd24fb79fe67dd05d7f7ccfa"}
Sep 15 19:04:22.725098 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:22.725078    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"
Sep 15 19:04:22.725201 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:22.725187    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7" event={"ID":"7ba7584f-beec-478c-b6f2-e2976ca9487a","Type":"ContainerStarted","Data":"c5cbf16860591e0c984de81eb1bf8f0c476112e1127021e79b17defd37a1d659"}
Sep 15 19:04:22.726411 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:22.726379    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp" event={"ID":"59bdbca8-25a1-486c-acdd-b2b2db5ab220","Type":"ContainerStarted","Data":"7af8ebd6ce0ea9615c9802a6ce0d391da6c9b801a4f1ff709a04f732c90c9fe2"}
Sep 15 19:04:22.726550 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:22.726433    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp" event={"ID":"59bdbca8-25a1-486c-acdd-b2b2db5ab220","Type":"ContainerStarted","Data":"43c82f034fdfa70509bbc8dd9ed65cf436e82830a1b53095273ed4a7db03c144"}
Sep 15 19:04:22.726792 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:22.726758    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"
Sep 15 19:04:22.743596 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:22.743554    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7" podStartSLOduration=1.743541505 podStartE2EDuration="1.743541505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:22.741261333 +0000 UTC m=+825.018030190" watchObservedRunningTime="2026-09-15 19:04:22.743541505 +0000 UTC m=+825.020310362"
Sep 15 19:04:22.757891 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:22.757853    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp" podStartSLOduration=1.7578432739999998 podStartE2EDuration="1.757843274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:22.755532992 +0000 UTC m=+825.032301848" watchObservedRunningTime="2026-09-15 19:04:22.757843274 +0000 UTC m=+825.034612133"
Sep 15 19:04:24.507669 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:24.507636    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:04:27.732200 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:27.732171    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:04:31.719510 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:31.719474    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"
Sep 15 19:04:32.729547 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:32.729293    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"
Sep 15 19:04:34.741531 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:34.741502    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"
Sep 15 19:04:34.741961 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:34.741582    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"
Sep 15 19:04:35.795359 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:35.795294    2550 generic.go:356] "Generic (PLEG): container finished" podID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" containerID="7aacf60a3dee30353f3fe29939cfdfbb253b997a13abe797c9d2dceaef9a9ec4" exitCode=0
Sep 15 19:04:35.795751 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:35.795358    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99" event={"ID":"ec73c8b1-223d-4859-8a85-d2faeabd20c9","Type":"ContainerDied","Data":"7aacf60a3dee30353f3fe29939cfdfbb253b997a13abe797c9d2dceaef9a9ec4"}
Sep 15 19:04:35.795904 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:35.795885    2550 scope.go:117] "RemoveContainer" containerID="7aacf60a3dee30353f3fe29939cfdfbb253b997a13abe797c9d2dceaef9a9ec4"
Sep 15 19:04:35.797300 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:35.797264    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:04:35.797438 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:35.797376    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:04:35.797438 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:35.797422    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:04:36.805774 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:36.805730    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99" event={"ID":"ec73c8b1-223d-4859-8a85-d2faeabd20c9","Type":"ContainerStarted","Data":"2889524a350cc2f423bba5ccbd7dfb0ef4492c2e3f1e049cb30559336142cc70"}
Sep 15 19:04:37.668863 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:37.668828    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:37.669051 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:37.669022    2550 patch_prober.go:28] interesting pod/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 15 19:04:37.669130 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:37.669080    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99" podUID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 15 19:04:38.376618 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.376581    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"]
Sep 15 19:04:38.379996 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.379974    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.384025 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.384003    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-08e9c4-openshift-default-dockercfg-qndmg\""
Sep 15 19:04:38.407397 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.407370    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"]
Sep 15 19:04:38.425521 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.425485    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-credential-socket\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.425682 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.425567    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-workload-certs\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.425682 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.425599    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-istio-envoy\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.425682 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.425629    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28fee392-71be-41d7-962a-9f17efffa9f4-istio-token\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.425682 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.425678    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28fee392-71be-41d7-962a-9f17efffa9f4-istiod-ca-cert\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.425912 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.425725    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-istio-data\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.425912 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.425760    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-workload-socket\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.425912 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.425812    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4prh\" (UniqueName: \"kubernetes.io/projected/28fee392-71be-41d7-962a-9f17efffa9f4-kube-api-access-g4prh\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.425912 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.425852    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28fee392-71be-41d7-962a-9f17efffa9f4-istio-podinfo\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.526746 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.526705    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4prh\" (UniqueName: \"kubernetes.io/projected/28fee392-71be-41d7-962a-9f17efffa9f4-kube-api-access-g4prh\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.526991 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.526967    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28fee392-71be-41d7-962a-9f17efffa9f4-istio-podinfo\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.527133 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.527026    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-credential-socket\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.527133 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.527092    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-workload-certs\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.527133 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.527120    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-istio-envoy\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.527293 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.527144    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28fee392-71be-41d7-962a-9f17efffa9f4-istio-token\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.527293 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.527191    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28fee392-71be-41d7-962a-9f17efffa9f4-istiod-ca-cert\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.527293 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.527239    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-istio-data\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.527293 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.527287    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-workload-socket\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.527691 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.527669    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-workload-socket\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.527921 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.527903    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-istio-data\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.528270 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.528246    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-credential-socket\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.528880 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.528861    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28fee392-71be-41d7-962a-9f17efffa9f4-istiod-ca-cert\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.528973 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.528760    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-workload-certs\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.530081 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.530058    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28fee392-71be-41d7-962a-9f17efffa9f4-istio-podinfo\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.530686 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.530668    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-istio-envoy\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.538224 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.538201    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4prh\" (UniqueName: \"kubernetes.io/projected/28fee392-71be-41d7-962a-9f17efffa9f4-kube-api-access-g4prh\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.538506 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.538490    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28fee392-71be-41d7-962a-9f17efffa9f4-istio-token\") pod \"e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") " pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:38.669559 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.669476    2550 patch_prober.go:28] interesting pod/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 15 19:04:38.669559 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.669529    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99" podUID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 15 19:04:38.725737 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:38.725700    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:39.018466 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:39.018377    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"]
Sep 15 19:04:39.022774 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:04:39.022739    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fee392_71be_41d7_962a_9f17efffa9f4.slice/crio-077f52da0a95f4d59e80b8dd89d960d6643f2201a086706969888765ca71ed92 WatchSource:0}: Error finding container 077f52da0a95f4d59e80b8dd89d960d6643f2201a086706969888765ca71ed92: Status 404 returned error can't find the container with id 077f52da0a95f4d59e80b8dd89d960d6643f2201a086706969888765ca71ed92
Sep 15 19:04:39.025712 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:39.025678    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:04:39.025821 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:39.025758    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:04:39.025821 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:39.025801    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:04:39.668833 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:39.668798    2550 patch_prober.go:28] interesting pod/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 15 19:04:39.669212 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:39.668866    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99" podUID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 15 19:04:39.829205 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:39.829170    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg" event={"ID":"28fee392-71be-41d7-962a-9f17efffa9f4","Type":"ContainerStarted","Data":"8d7d70e314a1936c593a54c96d4a63a62e36caf6b103b48b9087a8093fcbf5b4"}
Sep 15 19:04:39.829395 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:39.829220    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg" event={"ID":"28fee392-71be-41d7-962a-9f17efffa9f4","Type":"ContainerStarted","Data":"077f52da0a95f4d59e80b8dd89d960d6643f2201a086706969888765ca71ed92"}
Sep 15 19:04:39.849199 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:39.849153    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg" podStartSLOduration=1.849137712 podStartE2EDuration="1.849137712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:39.846103022 +0000 UTC m=+842.122871891" watchObservedRunningTime="2026-09-15 19:04:39.849137712 +0000 UTC m=+842.125906567"
Sep 15 19:04:40.669534 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:40.669499    2550 patch_prober.go:28] interesting pod/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 15 19:04:40.669892 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:40.669583    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99" podUID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 15 19:04:40.726214 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:40.726175    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:40.731304 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:40.731281    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:40.834229 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:40.834193    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:40.835165 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:40.835142    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:04:41.506624 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:41.506580    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:04:41.669488 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:41.669455    2550 patch_prober.go:28] interesting pod/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 15 19:04:41.669645 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:41.669511    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99" podUID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 15 19:04:42.673185 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:42.673155    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:42.673665 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:42.673578    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:42.674158 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:42.674139    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:04:44.277154 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:44.277116    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"]
Sep 15 19:04:44.279780 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:44.279764    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"
Sep 15 19:04:44.281823 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:44.281802    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-08e9c4\""
Sep 15 19:04:44.297954 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:44.297930    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"]
Sep 15 19:04:44.382092 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:44.382059    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744a9f07-be28-43b2-bb07-31bf4c6982f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq\" (UID: \"744a9f07-be28-43b2-bb07-31bf4c6982f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"
Sep 15 19:04:44.382286 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:44.382119    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsb9d\" (UniqueName: \"kubernetes.io/projected/744a9f07-be28-43b2-bb07-31bf4c6982f1-kube-api-access-dsb9d\") pod \"payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq\" (UID: \"744a9f07-be28-43b2-bb07-31bf4c6982f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"
Sep 15 19:04:44.483454 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:44.483405    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsb9d\" (UniqueName: \"kubernetes.io/projected/744a9f07-be28-43b2-bb07-31bf4c6982f1-kube-api-access-dsb9d\") pod \"payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq\" (UID: \"744a9f07-be28-43b2-bb07-31bf4c6982f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"
Sep 15 19:04:44.483636 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:44.483571    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744a9f07-be28-43b2-bb07-31bf4c6982f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq\" (UID: \"744a9f07-be28-43b2-bb07-31bf4c6982f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"
Sep 15 19:04:44.484492 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:44.484461    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744a9f07-be28-43b2-bb07-31bf4c6982f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq\" (UID: \"744a9f07-be28-43b2-bb07-31bf4c6982f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"
Sep 15 19:04:44.492244 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:44.492222    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsb9d\" (UniqueName: \"kubernetes.io/projected/744a9f07-be28-43b2-bb07-31bf4c6982f1-kube-api-access-dsb9d\") pod \"payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq\" (UID: \"744a9f07-be28-43b2-bb07-31bf4c6982f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"
Sep 15 19:04:44.592294 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:44.592237    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"
Sep 15 19:04:44.863338 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:44.863259    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"]
Sep 15 19:04:44.865879 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:04:44.865844    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744a9f07_be28_43b2_bb07_31bf4c6982f1.slice/crio-69cef105da687367d9d27d9ef7ee2a2028676b94a8fc48cd4cfeafe8a3afc231 WatchSource:0}: Error finding container 69cef105da687367d9d27d9ef7ee2a2028676b94a8fc48cd4cfeafe8a3afc231: Status 404 returned error can't find the container with id 69cef105da687367d9d27d9ef7ee2a2028676b94a8fc48cd4cfeafe8a3afc231
Sep 15 19:04:45.864060 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:45.864029    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq" event={"ID":"744a9f07-be28-43b2-bb07-31bf4c6982f1","Type":"ContainerStarted","Data":"574d7e11c130d54a08b4236997db2e921c1751549c341d9e5b373b070fa3f7cf"}
Sep 15 19:04:45.864451 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:45.864078    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq" event={"ID":"744a9f07-be28-43b2-bb07-31bf4c6982f1","Type":"ContainerStarted","Data":"69cef105da687367d9d27d9ef7ee2a2028676b94a8fc48cd4cfeafe8a3afc231"}
Sep 15 19:04:45.864451 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:45.864143    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"
Sep 15 19:04:45.878718 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:45.878668    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq" podStartSLOduration=1.878654729 podStartE2EDuration="1.878654729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:45.877775611 +0000 UTC m=+848.154544466" watchObservedRunningTime="2026-09-15 19:04:45.878654729 +0000 UTC m=+848.155423584"
Sep 15 19:04:53.710421 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:53.710384    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:04:56.870945 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:56.870917    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"
Sep 15 19:04:56.876453 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:56.876428    2550 patch_prober.go:28] interesting pod/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 19:04:56.876587 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:56.876489    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg" podUID="28fee392-71be-41d7-962a-9f17efffa9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 19:04:58.941665 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:58.941631    2550 generic.go:356] "Generic (PLEG): container finished" podID="28fee392-71be-41d7-962a-9f17efffa9f4" containerID="8d7d70e314a1936c593a54c96d4a63a62e36caf6b103b48b9087a8093fcbf5b4" exitCode=0
Sep 15 19:04:58.942056 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:58.941669    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg" event={"ID":"28fee392-71be-41d7-962a-9f17efffa9f4","Type":"ContainerDied","Data":"8d7d70e314a1936c593a54c96d4a63a62e36caf6b103b48b9087a8093fcbf5b4"}
Sep 15 19:04:58.942100 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:58.942070    2550 scope.go:117] "RemoveContainer" containerID="8d7d70e314a1936c593a54c96d4a63a62e36caf6b103b48b9087a8093fcbf5b4"
Sep 15 19:04:58.943150 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:58.943122    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:04:58.943213 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:58.943188    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:04:58.943247 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:58.943220    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:04:59.950888 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:04:59.950850    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg" event={"ID":"28fee392-71be-41d7-962a-9f17efffa9f4","Type":"ContainerStarted","Data":"d74537e9ccfd17645fe578331e208a2f8115bbb70cb6861ade6fb2ff36105812"}
Sep 15 19:05:00.726441 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:00.726394    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:05:00.726616 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:00.726585    2550 patch_prober.go:28] interesting pod/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 19:05:00.726668 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:00.726644    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg" podUID="28fee392-71be-41d7-962a-9f17efffa9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 19:05:01.726241 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:01.726205    2550 patch_prober.go:28] interesting pod/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 19:05:01.726640 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:01.726294    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg" podUID="28fee392-71be-41d7-962a-9f17efffa9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 19:05:02.726951 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:02.726913    2550 patch_prober.go:28] interesting pod/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 19:05:02.727369 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:02.726971    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg" podUID="28fee392-71be-41d7-962a-9f17efffa9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 19:05:03.727009 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:03.726964    2550 patch_prober.go:28] interesting pod/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 19:05:03.727422 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:03.727030    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg" podUID="28fee392-71be-41d7-962a-9f17efffa9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 19:05:04.726683 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:04.726643    2550 patch_prober.go:28] interesting pod/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 19:05:04.726851 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:04.726710    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg" podUID="28fee392-71be-41d7-962a-9f17efffa9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 19:05:05.729811 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:05.729782    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:05:05.730248 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:05.730235    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:05:05.731010 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:05.730987    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:05:07.409192 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:07.409153    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:05:21.307019 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:21.306984    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:05:44.914519 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:44.914482    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:05:50.598527 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:50.598493    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"]
Sep 15 19:05:50.601378 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:50.601361    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"
Sep 15 19:05:50.603537 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:50.603517    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e886bcb7-dockercfg-rdfmh\""
Sep 15 19:05:50.603641 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:50.603564    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e886bcb7\""
Sep 15 19:05:50.612467 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:50.612443    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"]
Sep 15 19:05:50.698166 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:50.698133    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7bc6019-23e7-452b-bcff-b2533b82ac33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8\" (UID: \"b7bc6019-23e7-452b-bcff-b2533b82ac33\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"
Sep 15 19:05:50.698339 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:50.698187    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln2pp\" (UniqueName: \"kubernetes.io/projected/b7bc6019-23e7-452b-bcff-b2533b82ac33-kube-api-access-ln2pp\") pod \"payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8\" (UID: \"b7bc6019-23e7-452b-bcff-b2533b82ac33\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"
Sep 15 19:05:50.798772 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:50.798743    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln2pp\" (UniqueName: \"kubernetes.io/projected/b7bc6019-23e7-452b-bcff-b2533b82ac33-kube-api-access-ln2pp\") pod \"payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8\" (UID: \"b7bc6019-23e7-452b-bcff-b2533b82ac33\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"
Sep 15 19:05:50.798919 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:50.798836    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7bc6019-23e7-452b-bcff-b2533b82ac33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8\" (UID: \"b7bc6019-23e7-452b-bcff-b2533b82ac33\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"
Sep 15 19:05:50.799478 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:50.799460    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7bc6019-23e7-452b-bcff-b2533b82ac33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8\" (UID: \"b7bc6019-23e7-452b-bcff-b2533b82ac33\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"
Sep 15 19:05:50.805425 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:50.805403    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln2pp\" (UniqueName: \"kubernetes.io/projected/b7bc6019-23e7-452b-bcff-b2533b82ac33-kube-api-access-ln2pp\") pod \"payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8\" (UID: \"b7bc6019-23e7-452b-bcff-b2533b82ac33\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"
Sep 15 19:05:50.914158 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:50.914094    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"
Sep 15 19:05:51.182142 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:51.182117    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"]
Sep 15 19:05:51.184695 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:05:51.184662    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7bc6019_23e7_452b_bcff_b2533b82ac33.slice/crio-5f533a8842a5130bd5ddd708461107b4b27e131b8802612a7180243aebc4c981 WatchSource:0}: Error finding container 5f533a8842a5130bd5ddd708461107b4b27e131b8802612a7180243aebc4c981: Status 404 returned error can't find the container with id 5f533a8842a5130bd5ddd708461107b4b27e131b8802612a7180243aebc4c981
Sep 15 19:05:51.206853 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:51.206827    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8" event={"ID":"b7bc6019-23e7-452b-bcff-b2533b82ac33","Type":"ContainerStarted","Data":"5f533a8842a5130bd5ddd708461107b4b27e131b8802612a7180243aebc4c981"}
Sep 15 19:05:52.212984 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:52.212949    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8" event={"ID":"b7bc6019-23e7-452b-bcff-b2533b82ac33","Type":"ContainerStarted","Data":"3566c767882aeacdd59e658260e59785b4de6a2e24dadd78366616aa3b3eb4b5"}
Sep 15 19:05:52.213409 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:52.213039    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"
Sep 15 19:05:52.227946 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:52.227904    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8" podStartSLOduration=2.227890992 podStartE2EDuration="2.227890992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:05:52.22720504 +0000 UTC m=+914.503973895" watchObservedRunningTime="2026-09-15 19:05:52.227890992 +0000 UTC m=+914.504659848"
Sep 15 19:05:58.714430 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:05:58.714394    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:06:03.220200 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:03.220171    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"
Sep 15 19:06:12.919773 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:12.919730    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:06:18.937492 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:18.937461    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"]
Sep 15 19:06:18.939927 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:18.939901    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:18.942099 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:18.942052    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-74ef1f\"/\"openshift-service-ca.crt\""
Sep 15 19:06:18.942234 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:18.942083    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-74ef1f\"/\"default-dockercfg-grz98\""
Sep 15 19:06:18.942234 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:18.942169    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-74ef1f\"/\"premium-w0-74ef1f-kserve-self-signed-certs\""
Sep 15 19:06:18.942234 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:18.942057    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-74ef1f\"/\"kube-root-ca.crt\""
Sep 15 19:06:18.949670 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:18.949648    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"]
Sep 15 19:06:19.058215 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.058177    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-dshm\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.058215 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.058209    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-tmp-dir\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.058457 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.058239    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9bj4\" (UniqueName: \"kubernetes.io/projected/626300c6-fa80-4ac5-973c-0546c076bdaa-kube-api-access-h9bj4\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.058457 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.058366    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-home\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.058457 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.058404    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-model-cache\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.058637 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.058467    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626300c6-fa80-4ac5-973c-0546c076bdaa-tls-certs\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.159306 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.159271    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-dshm\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.159306 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.159304    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-tmp-dir\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.159525 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.159355    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9bj4\" (UniqueName: \"kubernetes.io/projected/626300c6-fa80-4ac5-973c-0546c076bdaa-kube-api-access-h9bj4\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.159525 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.159384    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-home\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.159525 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.159404    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-model-cache\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.159525 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.159451    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626300c6-fa80-4ac5-973c-0546c076bdaa-tls-certs\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.159788 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.159759    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-tmp-dir\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.159908 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.159824    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-home\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.159908 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.159852    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-model-cache\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.161631 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.161610    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-dshm\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.161914 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.161895    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626300c6-fa80-4ac5-973c-0546c076bdaa-tls-certs\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.167196 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.167173    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9bj4\" (UniqueName: \"kubernetes.io/projected/626300c6-fa80-4ac5-973c-0546c076bdaa-kube-api-access-h9bj4\") pod \"premium-w0-74ef1f-kserve-57db786857-nfn5z\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") " pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.251264 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.251174    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:19.522114 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:19.522037    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"]
Sep 15 19:06:19.523410 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:06:19.523366    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod626300c6_fa80_4ac5_973c_0546c076bdaa.slice/crio-249b78dc63b2e9659a8516c5bb28ce4a1b6546d84e45b56355cbd37c4f9aeff5 WatchSource:0}: Error finding container 249b78dc63b2e9659a8516c5bb28ce4a1b6546d84e45b56355cbd37c4f9aeff5: Status 404 returned error can't find the container with id 249b78dc63b2e9659a8516c5bb28ce4a1b6546d84e45b56355cbd37c4f9aeff5
Sep 15 19:06:20.348654 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:20.348617    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z" event={"ID":"626300c6-fa80-4ac5-973c-0546c076bdaa","Type":"ContainerStarted","Data":"cea8bb640cf8a451f1d42ffa47a59df93e07651560e8b17b65e40635f685340d"}
Sep 15 19:06:20.349033 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:20.348660    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z" event={"ID":"626300c6-fa80-4ac5-973c-0546c076bdaa","Type":"ContainerStarted","Data":"249b78dc63b2e9659a8516c5bb28ce4a1b6546d84e45b56355cbd37c4f9aeff5"}
Sep 15 19:06:20.365333 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:20.365264    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z" podStartSLOduration=2.365249475 podStartE2EDuration="2.365249475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:06:20.363214651 +0000 UTC m=+942.639983506" watchObservedRunningTime="2026-09-15 19:06:20.365249475 +0000 UTC m=+942.642018370"
Sep 15 19:06:28.313772 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:28.313737    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:06:28.420664 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:28.420624    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"]
Sep 15 19:06:28.420940 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:28.420901    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8" podUID="b7bc6019-23e7-452b-bcff-b2533b82ac33" containerName="payload-pre-processing" containerID="cri-o://3566c767882aeacdd59e658260e59785b4de6a2e24dadd78366616aa3b3eb4b5" gracePeriod=30
Sep 15 19:06:28.695243 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:28.695215    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"
Sep 15 19:06:28.752899 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:28.752865    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln2pp\" (UniqueName: \"kubernetes.io/projected/b7bc6019-23e7-452b-bcff-b2533b82ac33-kube-api-access-ln2pp\") pod \"b7bc6019-23e7-452b-bcff-b2533b82ac33\" (UID: \"b7bc6019-23e7-452b-bcff-b2533b82ac33\") "
Sep 15 19:06:28.752899 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:28.752912    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7bc6019-23e7-452b-bcff-b2533b82ac33-plugins-config-volume\") pod \"b7bc6019-23e7-452b-bcff-b2533b82ac33\" (UID: \"b7bc6019-23e7-452b-bcff-b2533b82ac33\") "
Sep 15 19:06:28.753374 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:28.753343    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7bc6019-23e7-452b-bcff-b2533b82ac33-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7bc6019-23e7-452b-bcff-b2533b82ac33" (UID: "b7bc6019-23e7-452b-bcff-b2533b82ac33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:06:28.755262 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:28.755234    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bc6019-23e7-452b-bcff-b2533b82ac33-kube-api-access-ln2pp" (OuterVolumeSpecName: "kube-api-access-ln2pp") pod "b7bc6019-23e7-452b-bcff-b2533b82ac33" (UID: "b7bc6019-23e7-452b-bcff-b2533b82ac33"). InnerVolumeSpecName "kube-api-access-ln2pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:06:28.854005 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:28.853927    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7bc6019-23e7-452b-bcff-b2533b82ac33-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:06:28.854005 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:28.853955    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln2pp\" (UniqueName: \"kubernetes.io/projected/b7bc6019-23e7-452b-bcff-b2533b82ac33-kube-api-access-ln2pp\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:06:29.252018 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:29.251926    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:29.252303 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:29.252286    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:29.277819 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:29.277762    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:29.394834 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:29.394795    2550 generic.go:356] "Generic (PLEG): container finished" podID="b7bc6019-23e7-452b-bcff-b2533b82ac33" containerID="3566c767882aeacdd59e658260e59785b4de6a2e24dadd78366616aa3b3eb4b5" exitCode=0
Sep 15 19:06:29.395339 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:29.394859    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"
Sep 15 19:06:29.395339 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:29.394888    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8" event={"ID":"b7bc6019-23e7-452b-bcff-b2533b82ac33","Type":"ContainerDied","Data":"3566c767882aeacdd59e658260e59785b4de6a2e24dadd78366616aa3b3eb4b5"}
Sep 15 19:06:29.395339 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:29.394967    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8" event={"ID":"b7bc6019-23e7-452b-bcff-b2533b82ac33","Type":"ContainerDied","Data":"5f533a8842a5130bd5ddd708461107b4b27e131b8802612a7180243aebc4c981"}
Sep 15 19:06:29.395339 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:29.395020    2550 scope.go:117] "RemoveContainer" containerID="3566c767882aeacdd59e658260e59785b4de6a2e24dadd78366616aa3b3eb4b5"
Sep 15 19:06:29.416789 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:29.416760    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:06:29.418855 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:29.418806    2550 scope.go:117] "RemoveContainer" containerID="3566c767882aeacdd59e658260e59785b4de6a2e24dadd78366616aa3b3eb4b5"
Sep 15 19:06:29.419266 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:06:29.419236    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3566c767882aeacdd59e658260e59785b4de6a2e24dadd78366616aa3b3eb4b5\": container with ID starting with 3566c767882aeacdd59e658260e59785b4de6a2e24dadd78366616aa3b3eb4b5 not found: ID does not exist" containerID="3566c767882aeacdd59e658260e59785b4de6a2e24dadd78366616aa3b3eb4b5"
Sep 15 19:06:29.419390 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:29.419275    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3566c767882aeacdd59e658260e59785b4de6a2e24dadd78366616aa3b3eb4b5"} err="failed to get container status \"3566c767882aeacdd59e658260e59785b4de6a2e24dadd78366616aa3b3eb4b5\": rpc error: code = NotFound desc = could not find container \"3566c767882aeacdd59e658260e59785b4de6a2e24dadd78366616aa3b3eb4b5\": container with ID starting with 3566c767882aeacdd59e658260e59785b4de6a2e24dadd78366616aa3b3eb4b5 not found: ID does not exist"
Sep 15 19:06:29.422604 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:29.422582    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"]
Sep 15 19:06:29.428669 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:29.428634    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e886bcb7-5c94ff68bb-67bj8"]
Sep 15 19:06:30.295588 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:30.295554    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7bc6019-23e7-452b-bcff-b2533b82ac33" path="/var/lib/kubelet/pods/b7bc6019-23e7-452b-bcff-b2533b82ac33/volumes"
Sep 15 19:06:49.425423 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.425388    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"]
Sep 15 19:06:49.426035 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.426015    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7bc6019-23e7-452b-bcff-b2533b82ac33" containerName="payload-pre-processing"
Sep 15 19:06:49.426035 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.426036    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7bc6019-23e7-452b-bcff-b2533b82ac33" containerName="payload-pre-processing"
Sep 15 19:06:49.426189 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.426143    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7bc6019-23e7-452b-bcff-b2533b82ac33" containerName="payload-pre-processing"
Sep 15 19:06:49.430254 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.430235    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.432791 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.432768    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-08e9c4\"/\"default-dockercfg-c7x5f\""
Sep 15 19:06:49.432998 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.432821    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-08e9c4\"/\"test-model-08e9c4-kserve-self-signed-certs\""
Sep 15 19:06:49.433119 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.432829    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-08e9c4\"/\"kube-root-ca.crt\""
Sep 15 19:06:49.433395 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.433376    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-08e9c4\"/\"openshift-service-ca.crt\""
Sep 15 19:06:49.435025 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.435001    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"]
Sep 15 19:06:49.449677 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.449652    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-home\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.449782 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.449723    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-model-cache\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.449782 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.449761    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c30c381-76ff-4197-a53b-94e3af3722bc-tls-certs\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.449936 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.449800    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-dshm\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.449936 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.449840    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-tmp-dir\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.449936 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.449888    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfs5s\" (UniqueName: \"kubernetes.io/projected/2c30c381-76ff-4197-a53b-94e3af3722bc-kube-api-access-vfs5s\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.550835 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.550803    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-home\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.551010 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.550859    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-model-cache\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.551010 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.550881    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c30c381-76ff-4197-a53b-94e3af3722bc-tls-certs\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.551010 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.550905    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-dshm\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.551010 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.550923    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-tmp-dir\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.551010 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.550948    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfs5s\" (UniqueName: \"kubernetes.io/projected/2c30c381-76ff-4197-a53b-94e3af3722bc-kube-api-access-vfs5s\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.551261 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.551167    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-home\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.551364 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.551337    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-model-cache\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.551364 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.551358    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-tmp-dir\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.553364 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.553346    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-dshm\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.553528 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.553513    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c30c381-76ff-4197-a53b-94e3af3722bc-tls-certs\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.558239 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.558217    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfs5s\" (UniqueName: \"kubernetes.io/projected/2c30c381-76ff-4197-a53b-94e3af3722bc-kube-api-access-vfs5s\") pod \"test-model-08e9c4-kserve-54cf56c4b7-vqkrb\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") " pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.741735 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.741657    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:49.911762 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:49.911723    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:06:50.012498 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:50.012468    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"]
Sep 15 19:06:50.014254 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:06:50.014227    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c30c381_76ff_4197_a53b_94e3af3722bc.slice/crio-8eb4de33bb9ca7d7869074d03445f6b78e0299c3ce6210176536a66b9c7be841 WatchSource:0}: Error finding container 8eb4de33bb9ca7d7869074d03445f6b78e0299c3ce6210176536a66b9c7be841: Status 404 returned error can't find the container with id 8eb4de33bb9ca7d7869074d03445f6b78e0299c3ce6210176536a66b9c7be841
Sep 15 19:06:50.505662 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:50.505629    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb" event={"ID":"2c30c381-76ff-4197-a53b-94e3af3722bc","Type":"ContainerStarted","Data":"76bea33069c304d20854e3aca2b5f65bfad3a3d11046b5d45c2400181293ecfc"}
Sep 15 19:06:50.506109 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:50.505671    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb" event={"ID":"2c30c381-76ff-4197-a53b-94e3af3722bc","Type":"ContainerStarted","Data":"8eb4de33bb9ca7d7869074d03445f6b78e0299c3ce6210176536a66b9c7be841"}
Sep 15 19:06:50.521852 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:50.521807    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb" podStartSLOduration=1.521792477 podStartE2EDuration="1.521792477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:06:50.52015021 +0000 UTC m=+972.796919068" watchObservedRunningTime="2026-09-15 19:06:50.521792477 +0000 UTC m=+972.798561362"
Sep 15 19:06:59.742777 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:59.742736    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:59.743297 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:59.742797    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:06:59.755656 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:06:59.755609    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:07:00.564977 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:00.564905    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:07:07.414374 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:07.411078    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:07:24.511772 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:24.511741    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:07:46.647209 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.647171    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"]
Sep 15 19:07:46.652239 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.652213    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.655549 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.655520    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7c96ca\"/\"distinct-2-w3-7c96ca-kserve-self-signed-certs\""
Sep 15 19:07:46.655677 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.655552    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7c96ca\"/\"openshift-service-ca.crt\""
Sep 15 19:07:46.655677 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.655522    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7c96ca\"/\"kube-root-ca.crt\""
Sep 15 19:07:46.655862 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.655843    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7c96ca\"/\"default-dockercfg-pcpcj\""
Sep 15 19:07:46.662386 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.662362    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"]
Sep 15 19:07:46.668755 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.668727    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-dshm\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.668862 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.668797    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-model-cache\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.668919 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.668867    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-home\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.668974 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.668928    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-tmp-dir\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.668974 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.668950    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d2adc0c-e392-4097-b439-6305c4762543-tls-certs\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.669067 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.669007    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wrcw\" (UniqueName: \"kubernetes.io/projected/8d2adc0c-e392-4097-b439-6305c4762543-kube-api-access-9wrcw\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.770330 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.770289    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wrcw\" (UniqueName: \"kubernetes.io/projected/8d2adc0c-e392-4097-b439-6305c4762543-kube-api-access-9wrcw\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.770522 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.770382    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-dshm\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.770522 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.770434    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-model-cache\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.770522 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.770497    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-home\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.770657 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.770556    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-tmp-dir\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.770657 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.770577    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d2adc0c-e392-4097-b439-6305c4762543-tls-certs\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.770874 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.770847    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-model-cache\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.771002 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.770921    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-tmp-dir\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.771002 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.770945    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-home\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.772709 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.772689    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-dshm\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.772941 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.772924    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d2adc0c-e392-4097-b439-6305c4762543-tls-certs\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.777406 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.777385    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wrcw\" (UniqueName: \"kubernetes.io/projected/8d2adc0c-e392-4097-b439-6305c4762543-kube-api-access-9wrcw\") pod \"distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:46.963687 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:46.963603    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:47.248693 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:47.248616    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"]
Sep 15 19:07:47.251742 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:07:47.251717    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d2adc0c_e392_4097_b439_6305c4762543.slice/crio-99aa4cac820db09013b2d03917811e9bc9842c95e389261ad56a12160ca8c9ee WatchSource:0}: Error finding container 99aa4cac820db09013b2d03917811e9bc9842c95e389261ad56a12160ca8c9ee: Status 404 returned error can't find the container with id 99aa4cac820db09013b2d03917811e9bc9842c95e389261ad56a12160ca8c9ee
Sep 15 19:07:47.801595 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:47.801558    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq" event={"ID":"8d2adc0c-e392-4097-b439-6305c4762543","Type":"ContainerStarted","Data":"2f3f6b35973a18259a1e27127f58a6319a17ea18139f378fcacc00210c02d5a0"}
Sep 15 19:07:47.801969 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:47.801611    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq" event={"ID":"8d2adc0c-e392-4097-b439-6305c4762543","Type":"ContainerStarted","Data":"99aa4cac820db09013b2d03917811e9bc9842c95e389261ad56a12160ca8c9ee"}
Sep 15 19:07:47.819081 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:47.819033    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq" podStartSLOduration=1.8190204300000001 podStartE2EDuration="1.81902043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:47.817384942 +0000 UTC m=+1030.094153798" watchObservedRunningTime="2026-09-15 19:07:47.81902043 +0000 UTC m=+1030.095789285"
Sep 15 19:07:50.318717 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:50.318674    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:07:56.963965 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:56.963928    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:56.964569 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:56.964237    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:56.976835 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:56.976812    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:07:57.862738 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:07:57.862709    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:08:22.308012 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:08:22.307983    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:08:48.607519 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:08:48.607435    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:09:15.119812 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:15.119767    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"]
Sep 15 19:09:21.417911 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:21.417876    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:09:26.141123 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.141088    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"]
Sep 15 19:09:26.141609 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.141300    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7" podUID="7ba7584f-beec-478c-b6f2-e2976ca9487a" containerName="payload-processing" containerID="cri-o://82eb0db1b9b384cd4c4978b4850b55e2b690e27fdd24fb79fe67dd05d7f7ccfa" gracePeriod=30
Sep 15 19:09:26.146663 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.146635    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"]
Sep 15 19:09:26.146969 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.146829    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp" podUID="59bdbca8-25a1-486c-acdd-b2b2db5ab220" containerName="payload-pre-processing" containerID="cri-o://7af8ebd6ce0ea9615c9802a6ce0d391da6c9b801a4f1ff709a04f732c90c9fe2" gracePeriod=30
Sep 15 19:09:26.309256 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.309225    2550 generic.go:356] "Generic (PLEG): container finished" podID="7ba7584f-beec-478c-b6f2-e2976ca9487a" containerID="82eb0db1b9b384cd4c4978b4850b55e2b690e27fdd24fb79fe67dd05d7f7ccfa" exitCode=0
Sep 15 19:09:26.309435 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.309294    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7" event={"ID":"7ba7584f-beec-478c-b6f2-e2976ca9487a","Type":"ContainerDied","Data":"82eb0db1b9b384cd4c4978b4850b55e2b690e27fdd24fb79fe67dd05d7f7ccfa"}
Sep 15 19:09:26.311258 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.311230    2550 generic.go:356] "Generic (PLEG): container finished" podID="59bdbca8-25a1-486c-acdd-b2b2db5ab220" containerID="7af8ebd6ce0ea9615c9802a6ce0d391da6c9b801a4f1ff709a04f732c90c9fe2" exitCode=0
Sep 15 19:09:26.311400 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.311326    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp" event={"ID":"59bdbca8-25a1-486c-acdd-b2b2db5ab220","Type":"ContainerDied","Data":"7af8ebd6ce0ea9615c9802a6ce0d391da6c9b801a4f1ff709a04f732c90c9fe2"}
Sep 15 19:09:26.446503 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.446483    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"
Sep 15 19:09:26.449443 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.449427    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"
Sep 15 19:09:26.523506 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.523477    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba7584f-beec-478c-b6f2-e2976ca9487a-plugins-config-volume\") pod \"7ba7584f-beec-478c-b6f2-e2976ca9487a\" (UID: \"7ba7584f-beec-478c-b6f2-e2976ca9487a\") "
Sep 15 19:09:26.523651 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.523634    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz88b\" (UniqueName: \"kubernetes.io/projected/7ba7584f-beec-478c-b6f2-e2976ca9487a-kube-api-access-rz88b\") pod \"7ba7584f-beec-478c-b6f2-e2976ca9487a\" (UID: \"7ba7584f-beec-478c-b6f2-e2976ca9487a\") "
Sep 15 19:09:26.523848 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.523820    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba7584f-beec-478c-b6f2-e2976ca9487a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ba7584f-beec-478c-b6f2-e2976ca9487a" (UID: "7ba7584f-beec-478c-b6f2-e2976ca9487a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:26.525613 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.525593    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba7584f-beec-478c-b6f2-e2976ca9487a-kube-api-access-rz88b" (OuterVolumeSpecName: "kube-api-access-rz88b") pod "7ba7584f-beec-478c-b6f2-e2976ca9487a" (UID: "7ba7584f-beec-478c-b6f2-e2976ca9487a"). InnerVolumeSpecName "kube-api-access-rz88b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:26.624944 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.624915    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59bdbca8-25a1-486c-acdd-b2b2db5ab220-plugins-config-volume\") pod \"59bdbca8-25a1-486c-acdd-b2b2db5ab220\" (UID: \"59bdbca8-25a1-486c-acdd-b2b2db5ab220\") "
Sep 15 19:09:26.625102 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.625037    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfxnz\" (UniqueName: \"kubernetes.io/projected/59bdbca8-25a1-486c-acdd-b2b2db5ab220-kube-api-access-pfxnz\") pod \"59bdbca8-25a1-486c-acdd-b2b2db5ab220\" (UID: \"59bdbca8-25a1-486c-acdd-b2b2db5ab220\") "
Sep 15 19:09:26.625279 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.625254    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59bdbca8-25a1-486c-acdd-b2b2db5ab220-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "59bdbca8-25a1-486c-acdd-b2b2db5ab220" (UID: "59bdbca8-25a1-486c-acdd-b2b2db5ab220"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:26.625399 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.625384    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz88b\" (UniqueName: \"kubernetes.io/projected/7ba7584f-beec-478c-b6f2-e2976ca9487a-kube-api-access-rz88b\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:26.625458 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.625401    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba7584f-beec-478c-b6f2-e2976ca9487a-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:26.625458 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.625425    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59bdbca8-25a1-486c-acdd-b2b2db5ab220-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:26.627045 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.627025    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bdbca8-25a1-486c-acdd-b2b2db5ab220-kube-api-access-pfxnz" (OuterVolumeSpecName: "kube-api-access-pfxnz") pod "59bdbca8-25a1-486c-acdd-b2b2db5ab220" (UID: "59bdbca8-25a1-486c-acdd-b2b2db5ab220"). InnerVolumeSpecName "kube-api-access-pfxnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:26.726096 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:26.726021    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfxnz\" (UniqueName: \"kubernetes.io/projected/59bdbca8-25a1-486c-acdd-b2b2db5ab220-kube-api-access-pfxnz\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:27.323586 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:27.323545    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7" event={"ID":"7ba7584f-beec-478c-b6f2-e2976ca9487a","Type":"ContainerDied","Data":"c5cbf16860591e0c984de81eb1bf8f0c476112e1127021e79b17defd37a1d659"}
Sep 15 19:09:27.324072 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:27.323552    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"
Sep 15 19:09:27.324072 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:27.323657    2550 scope.go:117] "RemoveContainer" containerID="82eb0db1b9b384cd4c4978b4850b55e2b690e27fdd24fb79fe67dd05d7f7ccfa"
Sep 15 19:09:27.325124 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:27.325076    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"
Sep 15 19:09:27.325124 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:27.325091    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp" event={"ID":"59bdbca8-25a1-486c-acdd-b2b2db5ab220","Type":"ContainerDied","Data":"43c82f034fdfa70509bbc8dd9ed65cf436e82830a1b53095273ed4a7db03c144"}
Sep 15 19:09:27.333489 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:27.333472    2550 scope.go:117] "RemoveContainer" containerID="7af8ebd6ce0ea9615c9802a6ce0d391da6c9b801a4f1ff709a04f732c90c9fe2"
Sep 15 19:09:27.348000 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:27.347972    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"]
Sep 15 19:09:27.349897 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:27.349874    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5559a-757b5b5b47-clfs7"]
Sep 15 19:09:27.357754 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:27.357734    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"]
Sep 15 19:09:27.362912 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:27.362893    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5559a-78db7b4866-b9cgp"]
Sep 15 19:09:28.296091 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:28.296055    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59bdbca8-25a1-486c-acdd-b2b2db5ab220" path="/var/lib/kubelet/pods/59bdbca8-25a1-486c-acdd-b2b2db5ab220/volumes"
Sep 15 19:09:28.296583 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:28.296560    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ba7584f-beec-478c-b6f2-e2976ca9487a" path="/var/lib/kubelet/pods/7ba7584f-beec-478c-b6f2-e2976ca9487a/volumes"
Sep 15 19:09:30.139550 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.139507    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb" podUID="2c30c381-76ff-4197-a53b-94e3af3722bc" containerName="main" containerID="cri-o://76bea33069c304d20854e3aca2b5f65bfad3a3d11046b5d45c2400181293ecfc" gracePeriod=45
Sep 15 19:09:30.345039 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.345004    2550 generic.go:356] "Generic (PLEG): container finished" podID="2c30c381-76ff-4197-a53b-94e3af3722bc" containerID="76bea33069c304d20854e3aca2b5f65bfad3a3d11046b5d45c2400181293ecfc" exitCode=0
Sep 15 19:09:30.345180 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.345072    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb" event={"ID":"2c30c381-76ff-4197-a53b-94e3af3722bc","Type":"ContainerDied","Data":"76bea33069c304d20854e3aca2b5f65bfad3a3d11046b5d45c2400181293ecfc"}
Sep 15 19:09:30.400016 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.399950    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:09:30.560830 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.560802    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-home\") pod \"2c30c381-76ff-4197-a53b-94e3af3722bc\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") "
Sep 15 19:09:30.560989 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.560851    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c30c381-76ff-4197-a53b-94e3af3722bc-tls-certs\") pod \"2c30c381-76ff-4197-a53b-94e3af3722bc\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") "
Sep 15 19:09:30.560989 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.560878    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfs5s\" (UniqueName: \"kubernetes.io/projected/2c30c381-76ff-4197-a53b-94e3af3722bc-kube-api-access-vfs5s\") pod \"2c30c381-76ff-4197-a53b-94e3af3722bc\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") "
Sep 15 19:09:30.560989 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.560924    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-model-cache\") pod \"2c30c381-76ff-4197-a53b-94e3af3722bc\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") "
Sep 15 19:09:30.560989 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.560949    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-dshm\") pod \"2c30c381-76ff-4197-a53b-94e3af3722bc\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") "
Sep 15 19:09:30.561190 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.561048    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-tmp-dir\") pod \"2c30c381-76ff-4197-a53b-94e3af3722bc\" (UID: \"2c30c381-76ff-4197-a53b-94e3af3722bc\") "
Sep 15 19:09:30.561190 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.561103    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-home" (OuterVolumeSpecName: "home") pod "2c30c381-76ff-4197-a53b-94e3af3722bc" (UID: "2c30c381-76ff-4197-a53b-94e3af3722bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:30.561190 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.561179    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c30c381-76ff-4197-a53b-94e3af3722bc" (UID: "2c30c381-76ff-4197-a53b-94e3af3722bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:30.561335 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.561299    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c30c381-76ff-4197-a53b-94e3af3722bc" (UID: "2c30c381-76ff-4197-a53b-94e3af3722bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:30.561490 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.561467    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-model-cache\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:30.561490 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.561491    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-tmp-dir\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:30.561646 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.561505    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-home\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:30.563229 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.563195    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-dshm" (OuterVolumeSpecName: "dshm") pod "2c30c381-76ff-4197-a53b-94e3af3722bc" (UID: "2c30c381-76ff-4197-a53b-94e3af3722bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:30.563342 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.563276    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c30c381-76ff-4197-a53b-94e3af3722bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c30c381-76ff-4197-a53b-94e3af3722bc" (UID: "2c30c381-76ff-4197-a53b-94e3af3722bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:09:30.563342 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.563294    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c30c381-76ff-4197-a53b-94e3af3722bc-kube-api-access-vfs5s" (OuterVolumeSpecName: "kube-api-access-vfs5s") pod "2c30c381-76ff-4197-a53b-94e3af3722bc" (UID: "2c30c381-76ff-4197-a53b-94e3af3722bc"). InnerVolumeSpecName "kube-api-access-vfs5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:30.662023 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.661956    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c30c381-76ff-4197-a53b-94e3af3722bc-tls-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:30.662023 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.661979    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfs5s\" (UniqueName: \"kubernetes.io/projected/2c30c381-76ff-4197-a53b-94e3af3722bc-kube-api-access-vfs5s\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:30.662023 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:30.661989    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c30c381-76ff-4197-a53b-94e3af3722bc-dshm\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:31.351601 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:31.351558    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb" event={"ID":"2c30c381-76ff-4197-a53b-94e3af3722bc","Type":"ContainerDied","Data":"8eb4de33bb9ca7d7869074d03445f6b78e0299c3ce6210176536a66b9c7be841"}
Sep 15 19:09:31.352022 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:31.351614    2550 scope.go:117] "RemoveContainer" containerID="76bea33069c304d20854e3aca2b5f65bfad3a3d11046b5d45c2400181293ecfc"
Sep 15 19:09:31.352022 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:31.351572    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"
Sep 15 19:09:31.371690 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:31.371669    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"]
Sep 15 19:09:31.382868 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:31.382842    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-08e9c4/test-model-08e9c4-kserve-54cf56c4b7-vqkrb"]
Sep 15 19:09:32.296132 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:32.296101    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c30c381-76ff-4197-a53b-94e3af3722bc" path="/var/lib/kubelet/pods/2c30c381-76ff-4197-a53b-94e3af3722bc/volumes"
Sep 15 19:09:44.738086 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:44.738045    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"]
Sep 15 19:09:44.738595 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:44.738355    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq" podUID="744a9f07-be28-43b2-bb07-31bf4c6982f1" containerName="payload-pre-processing" containerID="cri-o://574d7e11c130d54a08b4236997db2e921c1751549c341d9e5b373b070fa3f7cf" gracePeriod=30
Sep 15 19:09:44.997003 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:44.996916    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"
Sep 15 19:09:45.090783 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:45.090746    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744a9f07-be28-43b2-bb07-31bf4c6982f1-plugins-config-volume\") pod \"744a9f07-be28-43b2-bb07-31bf4c6982f1\" (UID: \"744a9f07-be28-43b2-bb07-31bf4c6982f1\") "
Sep 15 19:09:45.090783 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:45.090790    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsb9d\" (UniqueName: \"kubernetes.io/projected/744a9f07-be28-43b2-bb07-31bf4c6982f1-kube-api-access-dsb9d\") pod \"744a9f07-be28-43b2-bb07-31bf4c6982f1\" (UID: \"744a9f07-be28-43b2-bb07-31bf4c6982f1\") "
Sep 15 19:09:45.091165 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:45.091135    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744a9f07-be28-43b2-bb07-31bf4c6982f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "744a9f07-be28-43b2-bb07-31bf4c6982f1" (UID: "744a9f07-be28-43b2-bb07-31bf4c6982f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:45.092978 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:45.092954    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744a9f07-be28-43b2-bb07-31bf4c6982f1-kube-api-access-dsb9d" (OuterVolumeSpecName: "kube-api-access-dsb9d") pod "744a9f07-be28-43b2-bb07-31bf4c6982f1" (UID: "744a9f07-be28-43b2-bb07-31bf4c6982f1"). InnerVolumeSpecName "kube-api-access-dsb9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:45.191651 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:45.191617    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744a9f07-be28-43b2-bb07-31bf4c6982f1-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:45.191651 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:45.191649    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsb9d\" (UniqueName: \"kubernetes.io/projected/744a9f07-be28-43b2-bb07-31bf4c6982f1-kube-api-access-dsb9d\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:45.439013 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:45.438978    2550 generic.go:356] "Generic (PLEG): container finished" podID="744a9f07-be28-43b2-bb07-31bf4c6982f1" containerID="574d7e11c130d54a08b4236997db2e921c1751549c341d9e5b373b070fa3f7cf" exitCode=0
Sep 15 19:09:45.439182 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:45.439020    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"
Sep 15 19:09:45.439182 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:45.439051    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq" event={"ID":"744a9f07-be28-43b2-bb07-31bf4c6982f1","Type":"ContainerDied","Data":"574d7e11c130d54a08b4236997db2e921c1751549c341d9e5b373b070fa3f7cf"}
Sep 15 19:09:45.439182 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:45.439089    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq" event={"ID":"744a9f07-be28-43b2-bb07-31bf4c6982f1","Type":"ContainerDied","Data":"69cef105da687367d9d27d9ef7ee2a2028676b94a8fc48cd4cfeafe8a3afc231"}
Sep 15 19:09:45.439182 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:45.439110    2550 scope.go:117] "RemoveContainer" containerID="574d7e11c130d54a08b4236997db2e921c1751549c341d9e5b373b070fa3f7cf"
Sep 15 19:09:45.448625 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:45.448603    2550 scope.go:117] "RemoveContainer" containerID="574d7e11c130d54a08b4236997db2e921c1751549c341d9e5b373b070fa3f7cf"
Sep 15 19:09:45.448896 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:09:45.448879    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574d7e11c130d54a08b4236997db2e921c1751549c341d9e5b373b070fa3f7cf\": container with ID starting with 574d7e11c130d54a08b4236997db2e921c1751549c341d9e5b373b070fa3f7cf not found: ID does not exist" containerID="574d7e11c130d54a08b4236997db2e921c1751549c341d9e5b373b070fa3f7cf"
Sep 15 19:09:45.448963 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:45.448904    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574d7e11c130d54a08b4236997db2e921c1751549c341d9e5b373b070fa3f7cf"} err="failed to get container status \"574d7e11c130d54a08b4236997db2e921c1751549c341d9e5b373b070fa3f7cf\": rpc error: code = NotFound desc = could not find container \"574d7e11c130d54a08b4236997db2e921c1751549c341d9e5b373b070fa3f7cf\": container with ID starting with 574d7e11c130d54a08b4236997db2e921c1751549c341d9e5b373b070fa3f7cf not found: ID does not exist"
Sep 15 19:09:45.460411 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:45.460388    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"]
Sep 15 19:09:45.463822 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:45.463798    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08e9c4-7d76f6b6bb-cpbsq"]
Sep 15 19:09:46.295991 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:46.295960    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="744a9f07-be28-43b2-bb07-31bf4c6982f1" path="/var/lib/kubelet/pods/744a9f07-be28-43b2-bb07-31bf4c6982f1/volumes"
Sep 15 19:09:51.314114 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:51.314077    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"]
Sep 15 19:09:51.314563 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:51.314362    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg" podUID="28fee392-71be-41d7-962a-9f17efffa9f4" containerName="istio-proxy" containerID="cri-o://d74537e9ccfd17645fe578331e208a2f8115bbb70cb6861ade6fb2ff36105812" gracePeriod=30
Sep 15 19:09:52.714083 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:52.714048    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:09:56.503543 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.503485    2550 generic.go:356] "Generic (PLEG): container finished" podID="28fee392-71be-41d7-962a-9f17efffa9f4" containerID="d74537e9ccfd17645fe578331e208a2f8115bbb70cb6861ade6fb2ff36105812" exitCode=0
Sep 15 19:09:56.503960 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.503577    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg" event={"ID":"28fee392-71be-41d7-962a-9f17efffa9f4","Type":"ContainerDied","Data":"d74537e9ccfd17645fe578331e208a2f8115bbb70cb6861ade6fb2ff36105812"}
Sep 15 19:09:56.503960 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.503739    2550 scope.go:117] "RemoveContainer" containerID="8d7d70e314a1936c593a54c96d4a63a62e36caf6b103b48b9087a8093fcbf5b4"
Sep 15 19:09:56.574189 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.574158    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:09:56.703041 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.702967    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-credential-socket\") pod \"28fee392-71be-41d7-962a-9f17efffa9f4\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") "
Sep 15 19:09:56.703041 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703012    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28fee392-71be-41d7-962a-9f17efffa9f4-istio-podinfo\") pod \"28fee392-71be-41d7-962a-9f17efffa9f4\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") "
Sep 15 19:09:56.703237 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703045    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-workload-certs\") pod \"28fee392-71be-41d7-962a-9f17efffa9f4\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") "
Sep 15 19:09:56.703237 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703076    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28fee392-71be-41d7-962a-9f17efffa9f4-istio-token\") pod \"28fee392-71be-41d7-962a-9f17efffa9f4\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") "
Sep 15 19:09:56.703237 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703109    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4prh\" (UniqueName: \"kubernetes.io/projected/28fee392-71be-41d7-962a-9f17efffa9f4-kube-api-access-g4prh\") pod \"28fee392-71be-41d7-962a-9f17efffa9f4\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") "
Sep 15 19:09:56.703237 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703139    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-istio-data\") pod \"28fee392-71be-41d7-962a-9f17efffa9f4\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") "
Sep 15 19:09:56.703419 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703273    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "28fee392-71be-41d7-962a-9f17efffa9f4" (UID: "28fee392-71be-41d7-962a-9f17efffa9f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:56.703419 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703351    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28fee392-71be-41d7-962a-9f17efffa9f4-istiod-ca-cert\") pod \"28fee392-71be-41d7-962a-9f17efffa9f4\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") "
Sep 15 19:09:56.703515 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703457    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-istio-envoy\") pod \"28fee392-71be-41d7-962a-9f17efffa9f4\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") "
Sep 15 19:09:56.703515 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703463    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "28fee392-71be-41d7-962a-9f17efffa9f4" (UID: "28fee392-71be-41d7-962a-9f17efffa9f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:56.703515 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703494    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-workload-socket\") pod \"28fee392-71be-41d7-962a-9f17efffa9f4\" (UID: \"28fee392-71be-41d7-962a-9f17efffa9f4\") "
Sep 15 19:09:56.703684 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703663    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "28fee392-71be-41d7-962a-9f17efffa9f4" (UID: "28fee392-71be-41d7-962a-9f17efffa9f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:56.703742 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703703    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fee392-71be-41d7-962a-9f17efffa9f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "28fee392-71be-41d7-962a-9f17efffa9f4" (UID: "28fee392-71be-41d7-962a-9f17efffa9f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:56.703908 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703878    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "28fee392-71be-41d7-962a-9f17efffa9f4" (UID: "28fee392-71be-41d7-962a-9f17efffa9f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:56.703978 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703933    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-credential-socket\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:56.703978 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703953    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-workload-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:56.703978 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703969    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-istio-data\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:56.704142 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.703984    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28fee392-71be-41d7-962a-9f17efffa9f4-istiod-ca-cert\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:56.705452 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.705419    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fee392-71be-41d7-962a-9f17efffa9f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "28fee392-71be-41d7-962a-9f17efffa9f4" (UID: "28fee392-71be-41d7-962a-9f17efffa9f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:56.705559 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.705479    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28fee392-71be-41d7-962a-9f17efffa9f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "28fee392-71be-41d7-962a-9f17efffa9f4" (UID: "28fee392-71be-41d7-962a-9f17efffa9f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:09:56.705559 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.705523    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fee392-71be-41d7-962a-9f17efffa9f4-kube-api-access-g4prh" (OuterVolumeSpecName: "kube-api-access-g4prh") pod "28fee392-71be-41d7-962a-9f17efffa9f4" (UID: "28fee392-71be-41d7-962a-9f17efffa9f4"). InnerVolumeSpecName "kube-api-access-g4prh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:56.705559 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.705525    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "28fee392-71be-41d7-962a-9f17efffa9f4" (UID: "28fee392-71be-41d7-962a-9f17efffa9f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:56.804575 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.804539    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28fee392-71be-41d7-962a-9f17efffa9f4-istio-podinfo\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:56.804575 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.804569    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28fee392-71be-41d7-962a-9f17efffa9f4-istio-token\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:56.804575 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.804579    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4prh\" (UniqueName: \"kubernetes.io/projected/28fee392-71be-41d7-962a-9f17efffa9f4-kube-api-access-g4prh\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:56.804804 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.804589    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-istio-envoy\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:56.804804 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:56.804598    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fee392-71be-41d7-962a-9f17efffa9f4-workload-socket\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:09:57.509512 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:57.509478    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg" event={"ID":"28fee392-71be-41d7-962a-9f17efffa9f4","Type":"ContainerDied","Data":"077f52da0a95f4d59e80b8dd89d960d6643f2201a086706969888765ca71ed92"}
Sep 15 19:09:57.509512 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:57.509508    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"
Sep 15 19:09:57.509988 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:57.509538    2550 scope.go:117] "RemoveContainer" containerID="d74537e9ccfd17645fe578331e208a2f8115bbb70cb6861ade6fb2ff36105812"
Sep 15 19:09:57.530185 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:57.530160    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"]
Sep 15 19:09:57.534712 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:57.534682    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-08e9c4-openshift-default-6785bc6c58-jwzxg"]
Sep 15 19:09:58.295853 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.295825    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28fee392-71be-41d7-962a-9f17efffa9f4" path="/var/lib/kubelet/pods/28fee392-71be-41d7-962a-9f17efffa9f4/volumes"
Sep 15 19:09:58.876707 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.876676    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"]
Sep 15 19:09:58.877089 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.877027    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28fee392-71be-41d7-962a-9f17efffa9f4" containerName="istio-proxy"
Sep 15 19:09:58.877089 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.877037    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="28fee392-71be-41d7-962a-9f17efffa9f4" containerName="istio-proxy"
Sep 15 19:09:58.877089 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.877051    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59bdbca8-25a1-486c-acdd-b2b2db5ab220" containerName="payload-pre-processing"
Sep 15 19:09:58.877089 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.877059    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="59bdbca8-25a1-486c-acdd-b2b2db5ab220" containerName="payload-pre-processing"
Sep 15 19:09:58.877089 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.877074    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ba7584f-beec-478c-b6f2-e2976ca9487a" containerName="payload-processing"
Sep 15 19:09:58.877089 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.877084    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ba7584f-beec-478c-b6f2-e2976ca9487a" containerName="payload-processing"
Sep 15 19:09:58.877342 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.877099    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744a9f07-be28-43b2-bb07-31bf4c6982f1" containerName="payload-pre-processing"
Sep 15 19:09:58.877342 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.877105    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="744a9f07-be28-43b2-bb07-31bf4c6982f1" containerName="payload-pre-processing"
Sep 15 19:09:58.877342 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.877152    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c30c381-76ff-4197-a53b-94e3af3722bc" containerName="main"
Sep 15 19:09:58.877342 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.877158    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c30c381-76ff-4197-a53b-94e3af3722bc" containerName="main"
Sep 15 19:09:58.877342 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.877221    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ba7584f-beec-478c-b6f2-e2976ca9487a" containerName="payload-processing"
Sep 15 19:09:58.877342 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.877229    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="28fee392-71be-41d7-962a-9f17efffa9f4" containerName="istio-proxy"
Sep 15 19:09:58.877342 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.877236    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c30c381-76ff-4197-a53b-94e3af3722bc" containerName="main"
Sep 15 19:09:58.877342 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.877243    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="744a9f07-be28-43b2-bb07-31bf4c6982f1" containerName="payload-pre-processing"
Sep 15 19:09:58.877342 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.877250    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="59bdbca8-25a1-486c-acdd-b2b2db5ab220" containerName="payload-pre-processing"
Sep 15 19:09:58.879853 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.879837    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"
Sep 15 19:09:58.882440 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.882417    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-7073b8\""
Sep 15 19:09:58.882565 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.882461    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-7073b8-dockercfg-fzb88\""
Sep 15 19:09:58.891294 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.891268    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"]
Sep 15 19:09:58.916222 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.916192    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"]
Sep 15 19:09:58.916645 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.916627    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28fee392-71be-41d7-962a-9f17efffa9f4" containerName="istio-proxy"
Sep 15 19:09:58.916734 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.916648    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="28fee392-71be-41d7-962a-9f17efffa9f4" containerName="istio-proxy"
Sep 15 19:09:58.916788 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.916774    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="28fee392-71be-41d7-962a-9f17efffa9f4" containerName="istio-proxy"
Sep 15 19:09:58.919794 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.919775    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"
Sep 15 19:09:58.930346 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:58.930324    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"]
Sep 15 19:09:59.024546 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.024511    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ab7060-e1b7-42ca-b61b-4f72653a13a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp\" (UID: \"44ab7060-e1b7-42ca-b61b-4f72653a13a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"
Sep 15 19:09:59.024546 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.024549    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr98d\" (UniqueName: \"kubernetes.io/projected/44ab7060-e1b7-42ca-b61b-4f72653a13a8-kube-api-access-sr98d\") pod \"payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp\" (UID: \"44ab7060-e1b7-42ca-b61b-4f72653a13a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"
Sep 15 19:09:59.024737 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.024658    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sglrn\" (UniqueName: \"kubernetes.io/projected/de73c1b5-7da7-4c3b-a75e-54c4368a9bc0-kube-api-access-sglrn\") pod \"payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6\" (UID: \"de73c1b5-7da7-4c3b-a75e-54c4368a9bc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"
Sep 15 19:09:59.024776 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.024739    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de73c1b5-7da7-4c3b-a75e-54c4368a9bc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6\" (UID: \"de73c1b5-7da7-4c3b-a75e-54c4368a9bc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"
Sep 15 19:09:59.125720 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.125686    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de73c1b5-7da7-4c3b-a75e-54c4368a9bc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6\" (UID: \"de73c1b5-7da7-4c3b-a75e-54c4368a9bc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"
Sep 15 19:09:59.125879 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.125739    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ab7060-e1b7-42ca-b61b-4f72653a13a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp\" (UID: \"44ab7060-e1b7-42ca-b61b-4f72653a13a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"
Sep 15 19:09:59.125879 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.125851    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr98d\" (UniqueName: \"kubernetes.io/projected/44ab7060-e1b7-42ca-b61b-4f72653a13a8-kube-api-access-sr98d\") pod \"payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp\" (UID: \"44ab7060-e1b7-42ca-b61b-4f72653a13a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"
Sep 15 19:09:59.125990 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.125931    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sglrn\" (UniqueName: \"kubernetes.io/projected/de73c1b5-7da7-4c3b-a75e-54c4368a9bc0-kube-api-access-sglrn\") pod \"payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6\" (UID: \"de73c1b5-7da7-4c3b-a75e-54c4368a9bc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"
Sep 15 19:09:59.126515 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.126487    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ab7060-e1b7-42ca-b61b-4f72653a13a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp\" (UID: \"44ab7060-e1b7-42ca-b61b-4f72653a13a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"
Sep 15 19:09:59.126515 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.126497    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de73c1b5-7da7-4c3b-a75e-54c4368a9bc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6\" (UID: \"de73c1b5-7da7-4c3b-a75e-54c4368a9bc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"
Sep 15 19:09:59.133367 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.133300    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr98d\" (UniqueName: \"kubernetes.io/projected/44ab7060-e1b7-42ca-b61b-4f72653a13a8-kube-api-access-sr98d\") pod \"payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp\" (UID: \"44ab7060-e1b7-42ca-b61b-4f72653a13a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"
Sep 15 19:09:59.133367 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.133363    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sglrn\" (UniqueName: \"kubernetes.io/projected/de73c1b5-7da7-4c3b-a75e-54c4368a9bc0-kube-api-access-sglrn\") pod \"payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6\" (UID: \"de73c1b5-7da7-4c3b-a75e-54c4368a9bc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"
Sep 15 19:09:59.191479 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.191456    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"
Sep 15 19:09:59.229791 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.229761    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"
Sep 15 19:09:59.465730 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.465705    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"]
Sep 15 19:09:59.512344 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.512304    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"]
Sep 15 19:09:59.514021 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:09:59.513999    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ab7060_e1b7_42ca_b61b_4f72653a13a8.slice/crio-00909c39363dcea57211720fabe4b6fc2c018bec4b56cd75391552f45e49197c WatchSource:0}: Error finding container 00909c39363dcea57211720fabe4b6fc2c018bec4b56cd75391552f45e49197c: Status 404 returned error can't find the container with id 00909c39363dcea57211720fabe4b6fc2c018bec4b56cd75391552f45e49197c
Sep 15 19:09:59.521528 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.521506    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp" event={"ID":"44ab7060-e1b7-42ca-b61b-4f72653a13a8","Type":"ContainerStarted","Data":"00909c39363dcea57211720fabe4b6fc2c018bec4b56cd75391552f45e49197c"}
Sep 15 19:09:59.523184 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:09:59.523163    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6" event={"ID":"de73c1b5-7da7-4c3b-a75e-54c4368a9bc0","Type":"ContainerStarted","Data":"6a3b9c5298386e046468157b7b743b46f0bdaf3a65b7d819d77f43cdb38f4b02"}
Sep 15 19:10:00.529991 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:00.529858    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp" event={"ID":"44ab7060-e1b7-42ca-b61b-4f72653a13a8","Type":"ContainerStarted","Data":"4871d5d88feb38cf77a81e70586439d35336f7d1b9c4a4b2f4c80286d0dcd2fc"}
Sep 15 19:10:00.529991 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:00.529925    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"
Sep 15 19:10:00.531433 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:00.531409    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6" event={"ID":"de73c1b5-7da7-4c3b-a75e-54c4368a9bc0","Type":"ContainerStarted","Data":"52fcfc6f8379aa6c162e93a875404a1fa667f5253012f0bd170452dcaef9610e"}
Sep 15 19:10:00.531546 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:00.531483    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"
Sep 15 19:10:00.545097 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:00.545056    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp" podStartSLOduration=2.545043538 podStartE2EDuration="2.545043538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:10:00.544165679 +0000 UTC m=+1162.820934536" watchObservedRunningTime="2026-09-15 19:10:00.545043538 +0000 UTC m=+1162.821812395"
Sep 15 19:10:00.559299 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:00.559250    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6" podStartSLOduration=2.559232918 podStartE2EDuration="2.559232918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:10:00.557638665 +0000 UTC m=+1162.834407522" watchObservedRunningTime="2026-09-15 19:10:00.559232918 +0000 UTC m=+1162.836001780"
Sep 15 19:10:11.538206 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:11.538175    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"
Sep 15 19:10:11.538771 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:11.538368    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"
Sep 15 19:10:20.117364 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.117326    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:10:20.321117 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.321040    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"]
Sep 15 19:10:20.325276 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.325252    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.327982 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.327957    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-33ff04-openshift-default-dockercfg-hl4pp\""
Sep 15 19:10:20.328276 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.328249    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcz6c\" (UniqueName: \"kubernetes.io/projected/9419198c-cfee-455c-8903-2cd4763e2e6e-kube-api-access-vcz6c\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.328468 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.328290    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9419198c-cfee-455c-8903-2cd4763e2e6e-istiod-ca-cert\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.328468 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.328427    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-envoy\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.328468 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.328460    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-podinfo\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.328641 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.328493    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-data\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.328641 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.328570    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-token\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.328758 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.328662    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-credential-socket\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.328758 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.328707    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-workload-certs\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.328758 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.328742    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-workload-socket\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.341903 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.341874    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"]
Sep 15 19:10:20.429805 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.429694    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-credential-socket\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.429805 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.429767    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-workload-certs\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.429805 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.429803    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-workload-socket\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.430100 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.429870    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcz6c\" (UniqueName: \"kubernetes.io/projected/9419198c-cfee-455c-8903-2cd4763e2e6e-kube-api-access-vcz6c\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.430100 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.429901    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9419198c-cfee-455c-8903-2cd4763e2e6e-istiod-ca-cert\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.430100 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.429930    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-envoy\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.430100 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.429961    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-podinfo\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.430100 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.429994    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-data\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.430100 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.430040    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-token\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.430443 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.430147    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-credential-socket\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.430853 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.430827    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9419198c-cfee-455c-8903-2cd4763e2e6e-istiod-ca-cert\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.431818 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.431629    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-data\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.431818 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.431770    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-workload-certs\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.432013 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.431902    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-workload-socket\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.433870 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.433561    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-envoy\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.433870 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.433771    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-podinfo\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.438697 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.438621    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-token\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.438875 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.438828    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcz6c\" (UniqueName: \"kubernetes.io/projected/9419198c-cfee-455c-8903-2cd4763e2e6e-kube-api-access-vcz6c\") pod \"e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") " pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.638514 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.638483    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:20.911568 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.911540    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"]
Sep 15 19:10:20.914044 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:10:20.914015    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9419198c_cfee_455c_8903_2cd4763e2e6e.slice/crio-7aa4fd760514ac86a0eb5414919dcb8f37ee6dfc355a62571819cbf0b4974333 WatchSource:0}: Error finding container 7aa4fd760514ac86a0eb5414919dcb8f37ee6dfc355a62571819cbf0b4974333: Status 404 returned error can't find the container with id 7aa4fd760514ac86a0eb5414919dcb8f37ee6dfc355a62571819cbf0b4974333
Sep 15 19:10:20.916762 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.916724    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:10:20.916880 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.916858    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:10:20.916929 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:20.916912    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:10:21.640060 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:21.640013    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l" event={"ID":"9419198c-cfee-455c-8903-2cd4763e2e6e","Type":"ContainerStarted","Data":"941622cd9ac6d4d765876b5bb5a2acda89fd852bbc33e33d2cec9a8b5e8132b4"}
Sep 15 19:10:21.640505 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:21.640119    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l" event={"ID":"9419198c-cfee-455c-8903-2cd4763e2e6e","Type":"ContainerStarted","Data":"7aa4fd760514ac86a0eb5414919dcb8f37ee6dfc355a62571819cbf0b4974333"}
Sep 15 19:10:21.657050 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:21.656942    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l" podStartSLOduration=1.656925072 podStartE2EDuration="1.656925072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:10:21.656285414 +0000 UTC m=+1183.933054270" watchObservedRunningTime="2026-09-15 19:10:21.656925072 +0000 UTC m=+1183.933693928"
Sep 15 19:10:22.639072 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:22.639037    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:22.644719 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:22.644692    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:22.645054 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:22.644995    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:22.645672 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:22.645654    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:26.390890 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:26.390849    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"]
Sep 15 19:10:26.394916 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:26.394893    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"
Sep 15 19:10:26.397223 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:26.397192    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-33ff04\""
Sep 15 19:10:26.397352 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:26.397198    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-33ff04-dockercfg-lgbgp\""
Sep 15 19:10:26.403028 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:26.402862    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"]
Sep 15 19:10:26.493688 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:26.493652    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5tj9\" (UniqueName: \"kubernetes.io/projected/b346cff5-cdbd-430d-acbd-571362ef7fea-kube-api-access-h5tj9\") pod \"payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64\" (UID: \"b346cff5-cdbd-430d-acbd-571362ef7fea\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"
Sep 15 19:10:26.493856 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:26.493820    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b346cff5-cdbd-430d-acbd-571362ef7fea-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64\" (UID: \"b346cff5-cdbd-430d-acbd-571362ef7fea\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"
Sep 15 19:10:26.595143 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:26.595099    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b346cff5-cdbd-430d-acbd-571362ef7fea-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64\" (UID: \"b346cff5-cdbd-430d-acbd-571362ef7fea\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"
Sep 15 19:10:26.595298 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:26.595233    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5tj9\" (UniqueName: \"kubernetes.io/projected/b346cff5-cdbd-430d-acbd-571362ef7fea-kube-api-access-h5tj9\") pod \"payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64\" (UID: \"b346cff5-cdbd-430d-acbd-571362ef7fea\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"
Sep 15 19:10:26.595882 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:26.595862    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b346cff5-cdbd-430d-acbd-571362ef7fea-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64\" (UID: \"b346cff5-cdbd-430d-acbd-571362ef7fea\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"
Sep 15 19:10:26.603084 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:26.603059    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5tj9\" (UniqueName: \"kubernetes.io/projected/b346cff5-cdbd-430d-acbd-571362ef7fea-kube-api-access-h5tj9\") pod \"payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64\" (UID: \"b346cff5-cdbd-430d-acbd-571362ef7fea\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"
Sep 15 19:10:26.707179 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:26.707103    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"
Sep 15 19:10:26.985072 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:26.985040    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"]
Sep 15 19:10:26.986286 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:10:26.986259    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb346cff5_cdbd_430d_acbd_571362ef7fea.slice/crio-3bedf0d67a74f4a282a07c7c9af59ff8f4f8ad6e6ef0da0884e3c701fbbdbaf5 WatchSource:0}: Error finding container 3bedf0d67a74f4a282a07c7c9af59ff8f4f8ad6e6ef0da0884e3c701fbbdbaf5: Status 404 returned error can't find the container with id 3bedf0d67a74f4a282a07c7c9af59ff8f4f8ad6e6ef0da0884e3c701fbbdbaf5
Sep 15 19:10:27.673000 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:27.672966    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64" event={"ID":"b346cff5-cdbd-430d-acbd-571362ef7fea","Type":"ContainerStarted","Data":"9fec8f459796aab49e1900dffe5efebe426c760093319b9cd191429a2a1e4312"}
Sep 15 19:10:27.673506 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:27.673017    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64" event={"ID":"b346cff5-cdbd-430d-acbd-571362ef7fea","Type":"ContainerStarted","Data":"3bedf0d67a74f4a282a07c7c9af59ff8f4f8ad6e6ef0da0884e3c701fbbdbaf5"}
Sep 15 19:10:27.673506 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:27.673081    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"
Sep 15 19:10:27.688494 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:27.688446    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64" podStartSLOduration=1.6884326120000002 podStartE2EDuration="1.688432612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:10:27.68713328 +0000 UTC m=+1189.963902136" watchObservedRunningTime="2026-09-15 19:10:27.688432612 +0000 UTC m=+1189.965201485"
Sep 15 19:10:38.681892 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:38.681854    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"
Sep 15 19:10:48.789145 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:48.789114    2550 generic.go:356] "Generic (PLEG): container finished" podID="9419198c-cfee-455c-8903-2cd4763e2e6e" containerID="941622cd9ac6d4d765876b5bb5a2acda89fd852bbc33e33d2cec9a8b5e8132b4" exitCode=0
Sep 15 19:10:48.789502 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:48.789188    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l" event={"ID":"9419198c-cfee-455c-8903-2cd4763e2e6e","Type":"ContainerDied","Data":"941622cd9ac6d4d765876b5bb5a2acda89fd852bbc33e33d2cec9a8b5e8132b4"}
Sep 15 19:10:48.789628 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:48.789613    2550 scope.go:117] "RemoveContainer" containerID="941622cd9ac6d4d765876b5bb5a2acda89fd852bbc33e33d2cec9a8b5e8132b4"
Sep 15 19:10:48.790749 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:48.790724    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:10:48.790815 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:48.790788    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:10:48.790851 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:48.790819    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:10:49.800441 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:49.800406    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l" event={"ID":"9419198c-cfee-455c-8903-2cd4763e2e6e","Type":"ContainerStarted","Data":"4ec2c7f5ef8233c5d45a1c87ff91d8f87ad45f2d407eccccac891d6dbf05dcd4"}
Sep 15 19:10:50.639665 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:50.639627    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:50.640719 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:50.640697    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 19:10:50.640824 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:50.640744    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l" podUID="9419198c-cfee-455c-8903-2cd4763e2e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 19:10:51.639652 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:51.639615    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 19:10:51.640050 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:51.639697    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l" podUID="9419198c-cfee-455c-8903-2cd4763e2e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 19:10:52.639850 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:52.639813    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 19:10:52.640234 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:52.639880    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l" podUID="9419198c-cfee-455c-8903-2cd4763e2e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 19:10:52.644908 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:52.644885    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:53.639086 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:53.639044    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 19:10:53.639264 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:53.639106    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l" podUID="9419198c-cfee-455c-8903-2cd4763e2e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 19:10:54.639383 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:54.639344    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 19:10:54.639782 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:54.639403    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l" podUID="9419198c-cfee-455c-8903-2cd4763e2e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 19:10:54.846895 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:54.846861    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8924-16ce30"]
Sep 15 19:10:54.850647 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:54.850626    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-16ce30"
Sep 15 19:10:54.852929 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:54.852906    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zrrnn\""
Sep 15 19:10:54.854979 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:54.854958    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8924-16ce30"]
Sep 15 19:10:54.876712 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:54.876683    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rlz7\" (UniqueName: \"kubernetes.io/projected/9add823e-a073-4c2a-94a5-0f013048ae21-kube-api-access-6rlz7\") pod \"test-curl-8924-16ce30\" (UID: \"9add823e-a073-4c2a-94a5-0f013048ae21\") " pod="opendatahub/test-curl-8924-16ce30"
Sep 15 19:10:54.977884 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:54.977787    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rlz7\" (UniqueName: \"kubernetes.io/projected/9add823e-a073-4c2a-94a5-0f013048ae21-kube-api-access-6rlz7\") pod \"test-curl-8924-16ce30\" (UID: \"9add823e-a073-4c2a-94a5-0f013048ae21\") " pod="opendatahub/test-curl-8924-16ce30"
Sep 15 19:10:54.985962 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:54.985934    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rlz7\" (UniqueName: \"kubernetes.io/projected/9add823e-a073-4c2a-94a5-0f013048ae21-kube-api-access-6rlz7\") pod \"test-curl-8924-16ce30\" (UID: \"9add823e-a073-4c2a-94a5-0f013048ae21\") " pod="opendatahub/test-curl-8924-16ce30"
Sep 15 19:10:55.163139 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:55.163101    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-16ce30"
Sep 15 19:10:55.436484 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:55.436460    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8924-16ce30"]
Sep 15 19:10:55.438978 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:10:55.438952    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9add823e_a073_4c2a_94a5_0f013048ae21.slice/crio-54acc7cd347b6aa24577514966f1b161cd598be485f5bc9db6d3f3e4de40ae99 WatchSource:0}: Error finding container 54acc7cd347b6aa24577514966f1b161cd598be485f5bc9db6d3f3e4de40ae99: Status 404 returned error can't find the container with id 54acc7cd347b6aa24577514966f1b161cd598be485f5bc9db6d3f3e4de40ae99
Sep 15 19:10:55.441444 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:55.441425    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:10:55.643782 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:55.643742    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:55.644937 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:55.644919    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:10:55.835756 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:55.835719    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8924-16ce30" event={"ID":"9add823e-a073-4c2a-94a5-0f013048ae21","Type":"ContainerStarted","Data":"54acc7cd347b6aa24577514966f1b161cd598be485f5bc9db6d3f3e4de40ae99"}
Sep 15 19:10:55.910369 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:55.910337    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:10:56.244741 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.244658    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"]
Sep 15 19:10:56.248743 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.248708    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.250972 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.250942    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-7073b8\"/\"openshift-service-ca.crt\""
Sep 15 19:10:56.251118 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.251001    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-7073b8\"/\"auth-test-model-7073b8-kserve-self-signed-certs\""
Sep 15 19:10:56.251297 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.251241    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-7073b8\"/\"default-dockercfg-44jww\""
Sep 15 19:10:56.251552 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.251499    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-7073b8\"/\"kube-root-ca.crt\""
Sep 15 19:10:56.255762 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.255734    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"]
Sep 15 19:10:56.288520 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.288476    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqdfd\" (UniqueName: \"kubernetes.io/projected/ab086655-be1e-406f-9d89-bc94b0c2aa56-kube-api-access-nqdfd\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.288726 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.288540    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-model-cache\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.288726 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.288618    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-home\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.288726 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.288673    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-dshm\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.288916 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.288743    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab086655-be1e-406f-9d89-bc94b0c2aa56-tls-certs\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.288916 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.288765    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-tmp-dir\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.389773 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.389730    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab086655-be1e-406f-9d89-bc94b0c2aa56-tls-certs\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.389977 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.389788    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-tmp-dir\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.389977 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.389869    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqdfd\" (UniqueName: \"kubernetes.io/projected/ab086655-be1e-406f-9d89-bc94b0c2aa56-kube-api-access-nqdfd\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.389977 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.389909    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-model-cache\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.389977 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.389958    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-home\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.390199 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.390014    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-dshm\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.390659 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.390609    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-tmp-dir\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.390790 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.390725    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-model-cache\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.390790 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.390772    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-home\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.392739 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.392707    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-dshm\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.393247 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.393228    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab086655-be1e-406f-9d89-bc94b0c2aa56-tls-certs\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.399516 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.399492    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqdfd\" (UniqueName: \"kubernetes.io/projected/ab086655-be1e-406f-9d89-bc94b0c2aa56-kube-api-access-nqdfd\") pod \"auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.562501 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.562460    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:10:56.842228 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:56.842202    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"]
Sep 15 19:10:57.373444 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:10:57.373407    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab086655_be1e_406f_9d89_bc94b0c2aa56.slice/crio-85cd2cab41819f2790f2c51a8e7d85747d0fcd20d7f800ea8f90c4a9f56f0a08 WatchSource:0}: Error finding container 85cd2cab41819f2790f2c51a8e7d85747d0fcd20d7f800ea8f90c4a9f56f0a08: Status 404 returned error can't find the container with id 85cd2cab41819f2790f2c51a8e7d85747d0fcd20d7f800ea8f90c4a9f56f0a08
Sep 15 19:10:57.850640 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:57.850602    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq" event={"ID":"ab086655-be1e-406f-9d89-bc94b0c2aa56","Type":"ContainerStarted","Data":"ba695e462fadadd202d8054b34a9e11a5a71dc366cadf63808517a46ed21c904"}
Sep 15 19:10:57.851099 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:57.850648    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq" event={"ID":"ab086655-be1e-406f-9d89-bc94b0c2aa56","Type":"ContainerStarted","Data":"85cd2cab41819f2790f2c51a8e7d85747d0fcd20d7f800ea8f90c4a9f56f0a08"}
Sep 15 19:10:57.852101 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:57.852074    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8924-16ce30" event={"ID":"9add823e-a073-4c2a-94a5-0f013048ae21","Type":"ContainerStarted","Data":"6faa2f3501b3a23971c10f60ee80b95d263da59552ef6125096c9b3a4ea2bd83"}
Sep 15 19:10:57.866185 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:57.866136    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq" podStartSLOduration=1.866123388 podStartE2EDuration="1.866123388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:10:57.86446195 +0000 UTC m=+1220.141230806" watchObservedRunningTime="2026-09-15 19:10:57.866123388 +0000 UTC m=+1220.142892243"
Sep 15 19:10:57.879764 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:57.879709    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8924-16ce30" podStartSLOduration=1.880434069 podStartE2EDuration="3.879691012s" totalImagesPullingTime="1.999256943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:10:57.877173255 +0000 UTC m=+1220.153942115" watchObservedRunningTime="2026-09-15 19:10:57.879691012 +0000 UTC m=+1220.156459868"
Sep 15 19:10:58.487792 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:58.487691    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8924-16ce30"]
Sep 15 19:10:58.490368 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:58.490329    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8924-16ce30"]
Sep 15 19:10:58.857489 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:10:58.857455    2550 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8924-16ce30" container="test-curl-8924-16ce30"
Sep 15 19:10:59.862615 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:10:59.862550    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8924-16ce30" podUID="9add823e-a073-4c2a-94a5-0f013048ae21" containerName="test-curl-8924-16ce30" containerID="cri-o://6faa2f3501b3a23971c10f60ee80b95d263da59552ef6125096c9b3a4ea2bd83" gracePeriod=2
Sep 15 19:10:59.864757 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:10:59.864721    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8924-16ce30\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="9add823e-a073-4c2a-94a5-0f013048ae21" pod="opendatahub/test-curl-8924-16ce30"
Sep 15 19:11:02.039795 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:02.039766    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-16ce30"
Sep 15 19:11:02.144007 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:02.143916    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rlz7\" (UniqueName: \"kubernetes.io/projected/9add823e-a073-4c2a-94a5-0f013048ae21-kube-api-access-6rlz7\") pod \"9add823e-a073-4c2a-94a5-0f013048ae21\" (UID: \"9add823e-a073-4c2a-94a5-0f013048ae21\") "
Sep 15 19:11:02.145985 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:02.145953    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9add823e-a073-4c2a-94a5-0f013048ae21-kube-api-access-6rlz7" (OuterVolumeSpecName: "kube-api-access-6rlz7") pod "9add823e-a073-4c2a-94a5-0f013048ae21" (UID: "9add823e-a073-4c2a-94a5-0f013048ae21"). InnerVolumeSpecName "kube-api-access-6rlz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:11:02.245382 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:02.245331    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rlz7\" (UniqueName: \"kubernetes.io/projected/9add823e-a073-4c2a-94a5-0f013048ae21-kube-api-access-6rlz7\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:11:02.296360 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:02.296304    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9add823e-a073-4c2a-94a5-0f013048ae21" path="/var/lib/kubelet/pods/9add823e-a073-4c2a-94a5-0f013048ae21/volumes"
Sep 15 19:11:02.879485 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:02.879451    2550 generic.go:356] "Generic (PLEG): container finished" podID="9add823e-a073-4c2a-94a5-0f013048ae21" containerID="6faa2f3501b3a23971c10f60ee80b95d263da59552ef6125096c9b3a4ea2bd83" exitCode=137
Sep 15 19:11:02.879663 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:02.879510    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-16ce30"
Sep 15 19:11:02.879663 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:02.879559    2550 scope.go:117] "RemoveContainer" containerID="6faa2f3501b3a23971c10f60ee80b95d263da59552ef6125096c9b3a4ea2bd83"
Sep 15 19:11:02.892895 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:02.892868    2550 scope.go:117] "RemoveContainer" containerID="6faa2f3501b3a23971c10f60ee80b95d263da59552ef6125096c9b3a4ea2bd83"
Sep 15 19:11:02.893420 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:11:02.893399    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6faa2f3501b3a23971c10f60ee80b95d263da59552ef6125096c9b3a4ea2bd83\": container with ID starting with 6faa2f3501b3a23971c10f60ee80b95d263da59552ef6125096c9b3a4ea2bd83 not found: ID does not exist" containerID="6faa2f3501b3a23971c10f60ee80b95d263da59552ef6125096c9b3a4ea2bd83"
Sep 15 19:11:02.893507 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:02.893428    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6faa2f3501b3a23971c10f60ee80b95d263da59552ef6125096c9b3a4ea2bd83"} err="failed to get container status \"6faa2f3501b3a23971c10f60ee80b95d263da59552ef6125096c9b3a4ea2bd83\": rpc error: code = NotFound desc = could not find container \"6faa2f3501b3a23971c10f60ee80b95d263da59552ef6125096c9b3a4ea2bd83\": container with ID starting with 6faa2f3501b3a23971c10f60ee80b95d263da59552ef6125096c9b3a4ea2bd83 not found: ID does not exist"
Sep 15 19:11:06.562818 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:06.562775    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:11:06.563354 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:06.562839    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:11:06.581182 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:06.581152    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:11:06.914535 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:06.914450    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:11:07.584155 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:07.584122    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"]
Sep 15 19:11:10.248637 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.248602    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"]
Sep 15 19:11:10.249180 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.249161    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9add823e-a073-4c2a-94a5-0f013048ae21" containerName="test-curl-8924-16ce30"
Sep 15 19:11:10.249268 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.249184    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9add823e-a073-4c2a-94a5-0f013048ae21" containerName="test-curl-8924-16ce30"
Sep 15 19:11:10.249363 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.249329    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9add823e-a073-4c2a-94a5-0f013048ae21" containerName="test-curl-8924-16ce30"
Sep 15 19:11:10.252466 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.252443    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"
Sep 15 19:11:10.254882 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.254860    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d2e6cc70-dockercfg-f92jt\""
Sep 15 19:11:10.254986 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.254933    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d2e6cc70\""
Sep 15 19:11:10.262330 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.262288    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"]
Sep 15 19:11:10.284874 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.284847    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"]
Sep 15 19:11:10.288956 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.288934    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"
Sep 15 19:11:10.298867 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.298844    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"]
Sep 15 19:11:10.325400 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.325361    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e0a64f-eeff-41bb-ad36-b0fee8a6896c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq\" (UID: \"e4e0a64f-eeff-41bb-ad36-b0fee8a6896c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"
Sep 15 19:11:10.325514 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.325425    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvj7q\" (UniqueName: \"kubernetes.io/projected/9433fdc9-33b2-4ab4-a148-59e559692414-kube-api-access-hvj7q\") pod \"payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9\" (UID: \"9433fdc9-33b2-4ab4-a148-59e559692414\") " pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"
Sep 15 19:11:10.325559 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.325518    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9433fdc9-33b2-4ab4-a148-59e559692414-plugins-config-volume\") pod \"payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9\" (UID: \"9433fdc9-33b2-4ab4-a148-59e559692414\") " pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"
Sep 15 19:11:10.325642 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.325621    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzx4s\" (UniqueName: \"kubernetes.io/projected/e4e0a64f-eeff-41bb-ad36-b0fee8a6896c-kube-api-access-qzx4s\") pod \"payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq\" (UID: \"e4e0a64f-eeff-41bb-ad36-b0fee8a6896c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"
Sep 15 19:11:10.426881 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.426843    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e0a64f-eeff-41bb-ad36-b0fee8a6896c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq\" (UID: \"e4e0a64f-eeff-41bb-ad36-b0fee8a6896c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"
Sep 15 19:11:10.427063 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.426898    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvj7q\" (UniqueName: \"kubernetes.io/projected/9433fdc9-33b2-4ab4-a148-59e559692414-kube-api-access-hvj7q\") pod \"payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9\" (UID: \"9433fdc9-33b2-4ab4-a148-59e559692414\") " pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"
Sep 15 19:11:10.427063 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.426976    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9433fdc9-33b2-4ab4-a148-59e559692414-plugins-config-volume\") pod \"payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9\" (UID: \"9433fdc9-33b2-4ab4-a148-59e559692414\") " pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"
Sep 15 19:11:10.427155 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.427078    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzx4s\" (UniqueName: \"kubernetes.io/projected/e4e0a64f-eeff-41bb-ad36-b0fee8a6896c-kube-api-access-qzx4s\") pod \"payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq\" (UID: \"e4e0a64f-eeff-41bb-ad36-b0fee8a6896c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"
Sep 15 19:11:10.427673 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.427648    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e0a64f-eeff-41bb-ad36-b0fee8a6896c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq\" (UID: \"e4e0a64f-eeff-41bb-ad36-b0fee8a6896c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"
Sep 15 19:11:10.427774 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.427701    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9433fdc9-33b2-4ab4-a148-59e559692414-plugins-config-volume\") pod \"payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9\" (UID: \"9433fdc9-33b2-4ab4-a148-59e559692414\") " pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"
Sep 15 19:11:10.437918 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.437885    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzx4s\" (UniqueName: \"kubernetes.io/projected/e4e0a64f-eeff-41bb-ad36-b0fee8a6896c-kube-api-access-qzx4s\") pod \"payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq\" (UID: \"e4e0a64f-eeff-41bb-ad36-b0fee8a6896c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"
Sep 15 19:11:10.439043 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.439024    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvj7q\" (UniqueName: \"kubernetes.io/projected/9433fdc9-33b2-4ab4-a148-59e559692414-kube-api-access-hvj7q\") pod \"payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9\" (UID: \"9433fdc9-33b2-4ab4-a148-59e559692414\") " pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"
Sep 15 19:11:10.564603 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.564568    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"
Sep 15 19:11:10.600975 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.600941    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"
Sep 15 19:11:10.836724 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.836686    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"]
Sep 15 19:11:10.838280 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:11:10.838252    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e0a64f_eeff_41bb_ad36_b0fee8a6896c.slice/crio-1899446da5422e3935f613311a92a560a0e29ed2716622839f47c9d2d8888fb0 WatchSource:0}: Error finding container 1899446da5422e3935f613311a92a560a0e29ed2716622839f47c9d2d8888fb0: Status 404 returned error can't find the container with id 1899446da5422e3935f613311a92a560a0e29ed2716622839f47c9d2d8888fb0
Sep 15 19:11:10.885514 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.885491    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"]
Sep 15 19:11:10.897406 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:11:10.897377    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9433fdc9_33b2_4ab4_a148_59e559692414.slice/crio-c3bd98918946fb2bacf44b5e03d6552bd5aaf86b5ad9c35b49c779cd6a217c46 WatchSource:0}: Error finding container c3bd98918946fb2bacf44b5e03d6552bd5aaf86b5ad9c35b49c779cd6a217c46: Status 404 returned error can't find the container with id c3bd98918946fb2bacf44b5e03d6552bd5aaf86b5ad9c35b49c779cd6a217c46
Sep 15 19:11:10.928153 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.928123    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9" event={"ID":"9433fdc9-33b2-4ab4-a148-59e559692414","Type":"ContainerStarted","Data":"c3bd98918946fb2bacf44b5e03d6552bd5aaf86b5ad9c35b49c779cd6a217c46"}
Sep 15 19:11:10.929817 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.929788    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq" event={"ID":"e4e0a64f-eeff-41bb-ad36-b0fee8a6896c","Type":"ContainerStarted","Data":"3a37f8acba0f55a1542c84c32a5e8cfb7217928afea12c6ab223763ef2025e57"}
Sep 15 19:11:10.929928 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.929848    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq" event={"ID":"e4e0a64f-eeff-41bb-ad36-b0fee8a6896c","Type":"ContainerStarted","Data":"1899446da5422e3935f613311a92a560a0e29ed2716622839f47c9d2d8888fb0"}
Sep 15 19:11:10.930018 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.929995    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"
Sep 15 19:11:10.947893 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:10.947805    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq" podStartSLOduration=0.947786469 podStartE2EDuration="947.786469ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:11:10.946640752 +0000 UTC m=+1233.223409636" watchObservedRunningTime="2026-09-15 19:11:10.947786469 +0000 UTC m=+1233.224555326"
Sep 15 19:11:11.936493 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:11.936447    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9" event={"ID":"9433fdc9-33b2-4ab4-a148-59e559692414","Type":"ContainerStarted","Data":"0430e3162b83b694cbd8f4704e07b569fc49dd5a59bc69d397c9fcd3f892fa5f"}
Sep 15 19:11:11.953521 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:11.953474    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9" podStartSLOduration=1.953460571 podStartE2EDuration="1.953460571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:11:11.951545164 +0000 UTC m=+1234.228314032" watchObservedRunningTime="2026-09-15 19:11:11.953460571 +0000 UTC m=+1234.230229467"
Sep 15 19:11:12.941059 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:12.941009    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"
Sep 15 19:11:18.480373 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:18.480338    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"]
Sep 15 19:11:18.480919 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:18.480569    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl" podUID="637a8186-5537-4288-9b91-44610151ec81" containerName="payload-pre-processing" containerID="cri-o://6438624ff3045fe266ddb1abcbfd2832bcf46cb95e05dc8a6a3280b1937b7dcc" gracePeriod=30
Sep 15 19:11:18.751932 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:18.751903    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"
Sep 15 19:11:18.904948 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:18.904908    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlrxw\" (UniqueName: \"kubernetes.io/projected/637a8186-5537-4288-9b91-44610151ec81-kube-api-access-nlrxw\") pod \"637a8186-5537-4288-9b91-44610151ec81\" (UID: \"637a8186-5537-4288-9b91-44610151ec81\") "
Sep 15 19:11:18.905123 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:18.905089    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637a8186-5537-4288-9b91-44610151ec81-plugins-config-volume\") pod \"637a8186-5537-4288-9b91-44610151ec81\" (UID: \"637a8186-5537-4288-9b91-44610151ec81\") "
Sep 15 19:11:18.905498 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:18.905469    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/637a8186-5537-4288-9b91-44610151ec81-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "637a8186-5537-4288-9b91-44610151ec81" (UID: "637a8186-5537-4288-9b91-44610151ec81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:11:18.907053 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:18.907022    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637a8186-5537-4288-9b91-44610151ec81-kube-api-access-nlrxw" (OuterVolumeSpecName: "kube-api-access-nlrxw") pod "637a8186-5537-4288-9b91-44610151ec81" (UID: "637a8186-5537-4288-9b91-44610151ec81"). InnerVolumeSpecName "kube-api-access-nlrxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:11:18.971136 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:18.971103    2550 generic.go:356] "Generic (PLEG): container finished" podID="637a8186-5537-4288-9b91-44610151ec81" containerID="6438624ff3045fe266ddb1abcbfd2832bcf46cb95e05dc8a6a3280b1937b7dcc" exitCode=0
Sep 15 19:11:18.971244 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:18.971148    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"
Sep 15 19:11:18.971244 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:18.971163    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl" event={"ID":"637a8186-5537-4288-9b91-44610151ec81","Type":"ContainerDied","Data":"6438624ff3045fe266ddb1abcbfd2832bcf46cb95e05dc8a6a3280b1937b7dcc"}
Sep 15 19:11:18.971244 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:18.971206    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl" event={"ID":"637a8186-5537-4288-9b91-44610151ec81","Type":"ContainerDied","Data":"ab3df66aa026b3d41b12966b25b941e8c21d24612aa203d4030daf0e52b10e1d"}
Sep 15 19:11:18.971244 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:18.971233    2550 scope.go:117] "RemoveContainer" containerID="6438624ff3045fe266ddb1abcbfd2832bcf46cb95e05dc8a6a3280b1937b7dcc"
Sep 15 19:11:18.980308 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:18.980271    2550 scope.go:117] "RemoveContainer" containerID="6438624ff3045fe266ddb1abcbfd2832bcf46cb95e05dc8a6a3280b1937b7dcc"
Sep 15 19:11:18.980654 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:11:18.980612    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6438624ff3045fe266ddb1abcbfd2832bcf46cb95e05dc8a6a3280b1937b7dcc\": container with ID starting with 6438624ff3045fe266ddb1abcbfd2832bcf46cb95e05dc8a6a3280b1937b7dcc not found: ID does not exist" containerID="6438624ff3045fe266ddb1abcbfd2832bcf46cb95e05dc8a6a3280b1937b7dcc"
Sep 15 19:11:18.980748 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:18.980655    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6438624ff3045fe266ddb1abcbfd2832bcf46cb95e05dc8a6a3280b1937b7dcc"} err="failed to get container status \"6438624ff3045fe266ddb1abcbfd2832bcf46cb95e05dc8a6a3280b1937b7dcc\": rpc error: code = NotFound desc = could not find container \"6438624ff3045fe266ddb1abcbfd2832bcf46cb95e05dc8a6a3280b1937b7dcc\": container with ID starting with 6438624ff3045fe266ddb1abcbfd2832bcf46cb95e05dc8a6a3280b1937b7dcc not found: ID does not exist"
Sep 15 19:11:18.995710 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:18.995655    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"]
Sep 15 19:11:19.000225 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:19.000204    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7c96ca-76bbd5f94-qnbtl"]
Sep 15 19:11:19.006688 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:19.006661    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637a8186-5537-4288-9b91-44610151ec81-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:11:19.006688 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:19.006685    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlrxw\" (UniqueName: \"kubernetes.io/projected/637a8186-5537-4288-9b91-44610151ec81-kube-api-access-nlrxw\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:11:20.296580 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:20.296545    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="637a8186-5537-4288-9b91-44610151ec81" path="/var/lib/kubelet/pods/637a8186-5537-4288-9b91-44610151ec81/volumes"
Sep 15 19:11:21.938448 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:21.938413    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"
Sep 15 19:11:22.603995 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:22.603954    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq" podUID="8d2adc0c-e392-4097-b439-6305c4762543" containerName="main" containerID="cri-o://2f3f6b35973a18259a1e27127f58a6319a17ea18139f378fcacc00210c02d5a0" gracePeriod=45
Sep 15 19:11:22.707648 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:22.707617    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:11:22.893749 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:22.893725    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:11:22.997991 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:22.997962    2550 generic.go:356] "Generic (PLEG): container finished" podID="8d2adc0c-e392-4097-b439-6305c4762543" containerID="2f3f6b35973a18259a1e27127f58a6319a17ea18139f378fcacc00210c02d5a0" exitCode=0
Sep 15 19:11:22.998480 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:22.998011    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"
Sep 15 19:11:22.998480 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:22.998043    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq" event={"ID":"8d2adc0c-e392-4097-b439-6305c4762543","Type":"ContainerDied","Data":"2f3f6b35973a18259a1e27127f58a6319a17ea18139f378fcacc00210c02d5a0"}
Sep 15 19:11:22.998480 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:22.998094    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq" event={"ID":"8d2adc0c-e392-4097-b439-6305c4762543","Type":"ContainerDied","Data":"99aa4cac820db09013b2d03917811e9bc9842c95e389261ad56a12160ca8c9ee"}
Sep 15 19:11:22.998480 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:22.998119    2550 scope.go:117] "RemoveContainer" containerID="2f3f6b35973a18259a1e27127f58a6319a17ea18139f378fcacc00210c02d5a0"
Sep 15 19:11:23.007531 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.007510    2550 scope.go:117] "RemoveContainer" containerID="2f3f6b35973a18259a1e27127f58a6319a17ea18139f378fcacc00210c02d5a0"
Sep 15 19:11:23.007846 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:11:23.007827    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3f6b35973a18259a1e27127f58a6319a17ea18139f378fcacc00210c02d5a0\": container with ID starting with 2f3f6b35973a18259a1e27127f58a6319a17ea18139f378fcacc00210c02d5a0 not found: ID does not exist" containerID="2f3f6b35973a18259a1e27127f58a6319a17ea18139f378fcacc00210c02d5a0"
Sep 15 19:11:23.007921 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.007853    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3f6b35973a18259a1e27127f58a6319a17ea18139f378fcacc00210c02d5a0"} err="failed to get container status \"2f3f6b35973a18259a1e27127f58a6319a17ea18139f378fcacc00210c02d5a0\": rpc error: code = NotFound desc = could not find container \"2f3f6b35973a18259a1e27127f58a6319a17ea18139f378fcacc00210c02d5a0\": container with ID starting with 2f3f6b35973a18259a1e27127f58a6319a17ea18139f378fcacc00210c02d5a0 not found: ID does not exist"
Sep 15 19:11:23.047454 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.047419    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-home\") pod \"8d2adc0c-e392-4097-b439-6305c4762543\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") "
Sep 15 19:11:23.047608 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.047520    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d2adc0c-e392-4097-b439-6305c4762543-tls-certs\") pod \"8d2adc0c-e392-4097-b439-6305c4762543\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") "
Sep 15 19:11:23.047608 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.047570    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-model-cache\") pod \"8d2adc0c-e392-4097-b439-6305c4762543\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") "
Sep 15 19:11:23.047608 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.047595    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-dshm\") pod \"8d2adc0c-e392-4097-b439-6305c4762543\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") "
Sep 15 19:11:23.047777 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.047655    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-tmp-dir\") pod \"8d2adc0c-e392-4097-b439-6305c4762543\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") "
Sep 15 19:11:23.047777 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.047683    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wrcw\" (UniqueName: \"kubernetes.io/projected/8d2adc0c-e392-4097-b439-6305c4762543-kube-api-access-9wrcw\") pod \"8d2adc0c-e392-4097-b439-6305c4762543\" (UID: \"8d2adc0c-e392-4097-b439-6305c4762543\") "
Sep 15 19:11:23.047777 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.047684    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-home" (OuterVolumeSpecName: "home") pod "8d2adc0c-e392-4097-b439-6305c4762543" (UID: "8d2adc0c-e392-4097-b439-6305c4762543"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:23.047992 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.047842    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d2adc0c-e392-4097-b439-6305c4762543" (UID: "8d2adc0c-e392-4097-b439-6305c4762543"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:23.047992 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.047882    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d2adc0c-e392-4097-b439-6305c4762543" (UID: "8d2adc0c-e392-4097-b439-6305c4762543"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:23.048273 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.048252    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-model-cache\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.048392 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.048276    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-tmp-dir\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.048392 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.048289    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-home\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.049870 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.049841    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-dshm" (OuterVolumeSpecName: "dshm") pod "8d2adc0c-e392-4097-b439-6305c4762543" (UID: "8d2adc0c-e392-4097-b439-6305c4762543"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:23.049978 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.049889    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2adc0c-e392-4097-b439-6305c4762543-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d2adc0c-e392-4097-b439-6305c4762543" (UID: "8d2adc0c-e392-4097-b439-6305c4762543"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:11:23.049978 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.049919    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2adc0c-e392-4097-b439-6305c4762543-kube-api-access-9wrcw" (OuterVolumeSpecName: "kube-api-access-9wrcw") pod "8d2adc0c-e392-4097-b439-6305c4762543" (UID: "8d2adc0c-e392-4097-b439-6305c4762543"). InnerVolumeSpecName "kube-api-access-9wrcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:11:23.149912 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.149823    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d2adc0c-e392-4097-b439-6305c4762543-dshm\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.149912 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.149869    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wrcw\" (UniqueName: \"kubernetes.io/projected/8d2adc0c-e392-4097-b439-6305c4762543-kube-api-access-9wrcw\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.149912 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.149886    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d2adc0c-e392-4097-b439-6305c4762543-tls-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.320601 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.320569    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"]
Sep 15 19:11:23.326921 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.326896    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/distinct-2-w3-7c96ca-kserve-564ccc9f5d-gpgrq"]
Sep 15 19:11:23.948240 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:23.948211    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"
Sep 15 19:11:24.297153 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:24.297118    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d2adc0c-e392-4097-b439-6305c4762543" path="/var/lib/kubelet/pods/8d2adc0c-e392-4097-b439-6305c4762543/volumes"
Sep 15 19:11:51.203557 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:11:51.203522    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:12:12.112459 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:12.112422    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:12:16.636065 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.636031    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"]
Sep 15 19:12:16.636672 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.636653    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="637a8186-5537-4288-9b91-44610151ec81" containerName="payload-pre-processing"
Sep 15 19:12:16.636719 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.636676    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="637a8186-5537-4288-9b91-44610151ec81" containerName="payload-pre-processing"
Sep 15 19:12:16.636719 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.636701    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d2adc0c-e392-4097-b439-6305c4762543" containerName="main"
Sep 15 19:12:16.636719 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.636709    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d2adc0c-e392-4097-b439-6305c4762543" containerName="main"
Sep 15 19:12:16.636834 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.636823    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d2adc0c-e392-4097-b439-6305c4762543" containerName="main"
Sep 15 19:12:16.636868 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.636841    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="637a8186-5537-4288-9b91-44610151ec81" containerName="payload-pre-processing"
Sep 15 19:12:16.640359 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.640338    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.642985 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.642967    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-74ef1f\"/\"embedding-w0-74ef1f-kserve-self-signed-certs\""
Sep 15 19:12:16.649543 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.649521    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"]
Sep 15 19:12:16.743323 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.743272    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-tmp-dir\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.743483 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.743329    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-home\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.743483 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.743388    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-dshm\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.743483 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.743457    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trsvl\" (UniqueName: \"kubernetes.io/projected/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-kube-api-access-trsvl\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.743648 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.743500    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-tls-certs\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.743648 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.743589    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-model-cache\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.844694 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.844655    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-model-cache\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.844883 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.844750    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-tmp-dir\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.844883 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.844782    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-home\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.844883 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.844796    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-dshm\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.844883 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.844815    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trsvl\" (UniqueName: \"kubernetes.io/projected/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-kube-api-access-trsvl\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.844883 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.844854    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-tls-certs\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.845155 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.845135    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-model-cache\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.845200 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.845185    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-tmp-dir\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.845235 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.845199    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-home\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.847085 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.847060    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-dshm\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.847242 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.847226    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-tls-certs\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.852007 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.851989    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trsvl\" (UniqueName: \"kubernetes.io/projected/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-kube-api-access-trsvl\") pod \"embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") " pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:16.950715 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:16.950656    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:17.219326 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:17.219285    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"]
Sep 15 19:12:17.220388 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:12:17.220361    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13718b72_1a1a_4cd5_a0a3_e32e991ca1f0.slice/crio-c75d7c7bf8e59e1a90073f513a7d886cc49d5d87d75bb79679af1eed608e37c6 WatchSource:0}: Error finding container c75d7c7bf8e59e1a90073f513a7d886cc49d5d87d75bb79679af1eed608e37c6: Status 404 returned error can't find the container with id c75d7c7bf8e59e1a90073f513a7d886cc49d5d87d75bb79679af1eed608e37c6
Sep 15 19:12:17.283157 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:17.283131    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w" event={"ID":"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0","Type":"ContainerStarted","Data":"c75d7c7bf8e59e1a90073f513a7d886cc49d5d87d75bb79679af1eed608e37c6"}
Sep 15 19:12:18.295802 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:18.295769    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w" event={"ID":"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0","Type":"ContainerStarted","Data":"478b1391b04b99bc8b3ec7067eb8bc48c149d0e086962cbcf16a20c9afa1b98a"}
Sep 15 19:12:18.308643 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:18.308595    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w" podStartSLOduration=2.30858047 podStartE2EDuration="2.30858047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:12:18.306997518 +0000 UTC m=+1300.583766365" watchObservedRunningTime="2026-09-15 19:12:18.30858047 +0000 UTC m=+1300.585349324"
Sep 15 19:12:26.951822 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:26.951787    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:26.952210 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:26.951839    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:26.963665 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:26.963642    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:27.343575 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:27.343550    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:12:39.805891 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:12:39.805855    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:13:01.105086 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:01.105003    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:13:23.697627 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:23.697596    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"]
Sep 15 19:13:23.698056 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:23.697812    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9" podUID="9433fdc9-33b2-4ab4-a148-59e559692414" containerName="payload-processing" containerID="cri-o://0430e3162b83b694cbd8f4704e07b569fc49dd5a59bc69d397c9fcd3f892fa5f" gracePeriod=30
Sep 15 19:13:23.706505 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:23.706277    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"]
Sep 15 19:13:23.706594 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:23.706534    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq" podUID="e4e0a64f-eeff-41bb-ad36-b0fee8a6896c" containerName="payload-pre-processing" containerID="cri-o://3a37f8acba0f55a1542c84c32a5e8cfb7217928afea12c6ab223763ef2025e57" gracePeriod=30
Sep 15 19:13:23.966370 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:23.966282    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"
Sep 15 19:13:24.005559 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.005532    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"
Sep 15 19:13:24.054208 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.054179    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzx4s\" (UniqueName: \"kubernetes.io/projected/e4e0a64f-eeff-41bb-ad36-b0fee8a6896c-kube-api-access-qzx4s\") pod \"e4e0a64f-eeff-41bb-ad36-b0fee8a6896c\" (UID: \"e4e0a64f-eeff-41bb-ad36-b0fee8a6896c\") "
Sep 15 19:13:24.054397 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.054226    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e0a64f-eeff-41bb-ad36-b0fee8a6896c-plugins-config-volume\") pod \"e4e0a64f-eeff-41bb-ad36-b0fee8a6896c\" (UID: \"e4e0a64f-eeff-41bb-ad36-b0fee8a6896c\") "
Sep 15 19:13:24.054397 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.054251    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvj7q\" (UniqueName: \"kubernetes.io/projected/9433fdc9-33b2-4ab4-a148-59e559692414-kube-api-access-hvj7q\") pod \"9433fdc9-33b2-4ab4-a148-59e559692414\" (UID: \"9433fdc9-33b2-4ab4-a148-59e559692414\") "
Sep 15 19:13:24.054397 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.054268    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9433fdc9-33b2-4ab4-a148-59e559692414-plugins-config-volume\") pod \"9433fdc9-33b2-4ab4-a148-59e559692414\" (UID: \"9433fdc9-33b2-4ab4-a148-59e559692414\") "
Sep 15 19:13:24.054663 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.054639    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e0a64f-eeff-41bb-ad36-b0fee8a6896c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4e0a64f-eeff-41bb-ad36-b0fee8a6896c" (UID: "e4e0a64f-eeff-41bb-ad36-b0fee8a6896c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:13:24.054720 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.054642    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9433fdc9-33b2-4ab4-a148-59e559692414-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9433fdc9-33b2-4ab4-a148-59e559692414" (UID: "9433fdc9-33b2-4ab4-a148-59e559692414"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:13:24.056292 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.056268    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e0a64f-eeff-41bb-ad36-b0fee8a6896c-kube-api-access-qzx4s" (OuterVolumeSpecName: "kube-api-access-qzx4s") pod "e4e0a64f-eeff-41bb-ad36-b0fee8a6896c" (UID: "e4e0a64f-eeff-41bb-ad36-b0fee8a6896c"). InnerVolumeSpecName "kube-api-access-qzx4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:13:24.056364 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.056294    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9433fdc9-33b2-4ab4-a148-59e559692414-kube-api-access-hvj7q" (OuterVolumeSpecName: "kube-api-access-hvj7q") pod "9433fdc9-33b2-4ab4-a148-59e559692414" (UID: "9433fdc9-33b2-4ab4-a148-59e559692414"). InnerVolumeSpecName "kube-api-access-hvj7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:13:24.155682 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.155646    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzx4s\" (UniqueName: \"kubernetes.io/projected/e4e0a64f-eeff-41bb-ad36-b0fee8a6896c-kube-api-access-qzx4s\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:13:24.155682 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.155677    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e0a64f-eeff-41bb-ad36-b0fee8a6896c-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:13:24.155865 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.155692    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvj7q\" (UniqueName: \"kubernetes.io/projected/9433fdc9-33b2-4ab4-a148-59e559692414-kube-api-access-hvj7q\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:13:24.155865 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.155701    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9433fdc9-33b2-4ab4-a148-59e559692414-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:13:24.620423 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.620351    2550 generic.go:356] "Generic (PLEG): container finished" podID="9433fdc9-33b2-4ab4-a148-59e559692414" containerID="0430e3162b83b694cbd8f4704e07b569fc49dd5a59bc69d397c9fcd3f892fa5f" exitCode=0
Sep 15 19:13:24.620423 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.620396    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"
Sep 15 19:13:24.620699 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.620439    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9" event={"ID":"9433fdc9-33b2-4ab4-a148-59e559692414","Type":"ContainerDied","Data":"0430e3162b83b694cbd8f4704e07b569fc49dd5a59bc69d397c9fcd3f892fa5f"}
Sep 15 19:13:24.620699 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.620496    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9" event={"ID":"9433fdc9-33b2-4ab4-a148-59e559692414","Type":"ContainerDied","Data":"c3bd98918946fb2bacf44b5e03d6552bd5aaf86b5ad9c35b49c779cd6a217c46"}
Sep 15 19:13:24.620699 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.620526    2550 scope.go:117] "RemoveContainer" containerID="0430e3162b83b694cbd8f4704e07b569fc49dd5a59bc69d397c9fcd3f892fa5f"
Sep 15 19:13:24.621842 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.621806    2550 generic.go:356] "Generic (PLEG): container finished" podID="e4e0a64f-eeff-41bb-ad36-b0fee8a6896c" containerID="3a37f8acba0f55a1542c84c32a5e8cfb7217928afea12c6ab223763ef2025e57" exitCode=0
Sep 15 19:13:24.621942 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.621836    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq" event={"ID":"e4e0a64f-eeff-41bb-ad36-b0fee8a6896c","Type":"ContainerDied","Data":"3a37f8acba0f55a1542c84c32a5e8cfb7217928afea12c6ab223763ef2025e57"}
Sep 15 19:13:24.621942 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.621854    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"
Sep 15 19:13:24.621942 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.621890    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq" event={"ID":"e4e0a64f-eeff-41bb-ad36-b0fee8a6896c","Type":"ContainerDied","Data":"1899446da5422e3935f613311a92a560a0e29ed2716622839f47c9d2d8888fb0"}
Sep 15 19:13:24.628895 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.628879    2550 scope.go:117] "RemoveContainer" containerID="0430e3162b83b694cbd8f4704e07b569fc49dd5a59bc69d397c9fcd3f892fa5f"
Sep 15 19:13:24.629133 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:13:24.629114    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0430e3162b83b694cbd8f4704e07b569fc49dd5a59bc69d397c9fcd3f892fa5f\": container with ID starting with 0430e3162b83b694cbd8f4704e07b569fc49dd5a59bc69d397c9fcd3f892fa5f not found: ID does not exist" containerID="0430e3162b83b694cbd8f4704e07b569fc49dd5a59bc69d397c9fcd3f892fa5f"
Sep 15 19:13:24.629195 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.629140    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0430e3162b83b694cbd8f4704e07b569fc49dd5a59bc69d397c9fcd3f892fa5f"} err="failed to get container status \"0430e3162b83b694cbd8f4704e07b569fc49dd5a59bc69d397c9fcd3f892fa5f\": rpc error: code = NotFound desc = could not find container \"0430e3162b83b694cbd8f4704e07b569fc49dd5a59bc69d397c9fcd3f892fa5f\": container with ID starting with 0430e3162b83b694cbd8f4704e07b569fc49dd5a59bc69d397c9fcd3f892fa5f not found: ID does not exist"
Sep 15 19:13:24.629195 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.629156    2550 scope.go:117] "RemoveContainer" containerID="3a37f8acba0f55a1542c84c32a5e8cfb7217928afea12c6ab223763ef2025e57"
Sep 15 19:13:24.636806 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.636791    2550 scope.go:117] "RemoveContainer" containerID="3a37f8acba0f55a1542c84c32a5e8cfb7217928afea12c6ab223763ef2025e57"
Sep 15 19:13:24.637047 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:13:24.637031    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a37f8acba0f55a1542c84c32a5e8cfb7217928afea12c6ab223763ef2025e57\": container with ID starting with 3a37f8acba0f55a1542c84c32a5e8cfb7217928afea12c6ab223763ef2025e57 not found: ID does not exist" containerID="3a37f8acba0f55a1542c84c32a5e8cfb7217928afea12c6ab223763ef2025e57"
Sep 15 19:13:24.637094 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.637052    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a37f8acba0f55a1542c84c32a5e8cfb7217928afea12c6ab223763ef2025e57"} err="failed to get container status \"3a37f8acba0f55a1542c84c32a5e8cfb7217928afea12c6ab223763ef2025e57\": rpc error: code = NotFound desc = could not find container \"3a37f8acba0f55a1542c84c32a5e8cfb7217928afea12c6ab223763ef2025e57\": container with ID starting with 3a37f8acba0f55a1542c84c32a5e8cfb7217928afea12c6ab223763ef2025e57 not found: ID does not exist"
Sep 15 19:13:24.665944 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.665922    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"]
Sep 15 19:13:24.672427 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.672407    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2e6cc70-85747db998-crqwq"]
Sep 15 19:13:24.692932 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.692908    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"]
Sep 15 19:13:24.703463 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.703445    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9"]
Sep 15 19:13:24.947018 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:24.946930    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d2e6cc70-7cf55bd589-6rxc9" podUID="9433fdc9-33b2-4ab4-a148-59e559692414" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.61:9004: i/o timeout"
Sep 15 19:13:26.296377 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:26.296307    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9433fdc9-33b2-4ab4-a148-59e559692414" path="/var/lib/kubelet/pods/9433fdc9-33b2-4ab4-a148-59e559692414/volumes"
Sep 15 19:13:26.296844 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:26.296714    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4e0a64f-eeff-41bb-ad36-b0fee8a6896c" path="/var/lib/kubelet/pods/e4e0a64f-eeff-41bb-ad36-b0fee8a6896c/volumes"
Sep 15 19:13:30.506419 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:30.506384    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:13:50.710923 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:50.710887    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:13:51.990300 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:51.990257    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"]
Sep 15 19:13:51.999118 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:13:51.999088    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"]
Sep 15 19:14:02.757661 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:02.757630    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"]
Sep 15 19:14:02.758116 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:02.757837    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv" podUID="0cc77959-6e75-41db-8767-858a6f6c9052" containerName="maas-api" containerID="cri-o://5d61df8d4fe609e48a0da4ff5f4e7b87f66a8402694d0e5264bd7373534c3cd0" gracePeriod=30
Sep 15 19:14:02.841591 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:02.841559    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"]
Sep 15 19:14:02.841851 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:02.841825    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp" podUID="e739f55c-0ca9-442e-b6d0-dc8144acafd0" containerName="payload-pre-processing" containerID="cri-o://a9db53c8dd3afe9bbded6d21a35afb2d6dd727659a2559777451eaf238b5e717" gracePeriod=30
Sep 15 19:14:03.107809 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.107779    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:14:03.126704 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.126571    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"
Sep 15 19:14:03.231166 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.231134    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mfww\" (UniqueName: \"kubernetes.io/projected/0cc77959-6e75-41db-8767-858a6f6c9052-kube-api-access-7mfww\") pod \"0cc77959-6e75-41db-8767-858a6f6c9052\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") "
Sep 15 19:14:03.231363 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.231209    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc77959-6e75-41db-8767-858a6f6c9052-openshift-service-ca\") pod \"0cc77959-6e75-41db-8767-858a6f6c9052\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") "
Sep 15 19:14:03.231363 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.231232    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psq79\" (UniqueName: \"kubernetes.io/projected/e739f55c-0ca9-442e-b6d0-dc8144acafd0-kube-api-access-psq79\") pod \"e739f55c-0ca9-442e-b6d0-dc8144acafd0\" (UID: \"e739f55c-0ca9-442e-b6d0-dc8144acafd0\") "
Sep 15 19:14:03.231363 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.231271    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e739f55c-0ca9-442e-b6d0-dc8144acafd0-plugins-config-volume\") pod \"e739f55c-0ca9-442e-b6d0-dc8144acafd0\" (UID: \"e739f55c-0ca9-442e-b6d0-dc8144acafd0\") "
Sep 15 19:14:03.231363 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.231350    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cc77959-6e75-41db-8767-858a6f6c9052-metrics-certs\") pod \"0cc77959-6e75-41db-8767-858a6f6c9052\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") "
Sep 15 19:14:03.231611 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.231373    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc77959-6e75-41db-8767-858a6f6c9052-maas-api-tls\") pod \"0cc77959-6e75-41db-8767-858a6f6c9052\" (UID: \"0cc77959-6e75-41db-8767-858a6f6c9052\") "
Sep 15 19:14:03.231611 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.231575    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc77959-6e75-41db-8767-858a6f6c9052-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0cc77959-6e75-41db-8767-858a6f6c9052" (UID: "0cc77959-6e75-41db-8767-858a6f6c9052"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:14:03.231728 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.231690    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e739f55c-0ca9-442e-b6d0-dc8144acafd0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e739f55c-0ca9-442e-b6d0-dc8144acafd0" (UID: "e739f55c-0ca9-442e-b6d0-dc8144acafd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:14:03.233347 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.233290    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e739f55c-0ca9-442e-b6d0-dc8144acafd0-kube-api-access-psq79" (OuterVolumeSpecName: "kube-api-access-psq79") pod "e739f55c-0ca9-442e-b6d0-dc8144acafd0" (UID: "e739f55c-0ca9-442e-b6d0-dc8144acafd0"). InnerVolumeSpecName "kube-api-access-psq79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:03.233471 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.233450    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc77959-6e75-41db-8767-858a6f6c9052-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0cc77959-6e75-41db-8767-858a6f6c9052" (UID: "0cc77959-6e75-41db-8767-858a6f6c9052"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:14:03.233538 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.233476    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc77959-6e75-41db-8767-858a6f6c9052-kube-api-access-7mfww" (OuterVolumeSpecName: "kube-api-access-7mfww") pod "0cc77959-6e75-41db-8767-858a6f6c9052" (UID: "0cc77959-6e75-41db-8767-858a6f6c9052"). InnerVolumeSpecName "kube-api-access-7mfww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:03.233538 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.233505    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc77959-6e75-41db-8767-858a6f6c9052-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0cc77959-6e75-41db-8767-858a6f6c9052" (UID: "0cc77959-6e75-41db-8767-858a6f6c9052"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:14:03.333021 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.332945    2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cc77959-6e75-41db-8767-858a6f6c9052-metrics-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:03.333021 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.332974    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc77959-6e75-41db-8767-858a6f6c9052-maas-api-tls\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:03.333021 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.332985    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mfww\" (UniqueName: \"kubernetes.io/projected/0cc77959-6e75-41db-8767-858a6f6c9052-kube-api-access-7mfww\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:03.333021 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.332994    2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc77959-6e75-41db-8767-858a6f6c9052-openshift-service-ca\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:03.333021 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.333003    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psq79\" (UniqueName: \"kubernetes.io/projected/e739f55c-0ca9-442e-b6d0-dc8144acafd0-kube-api-access-psq79\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:03.333021 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.333012    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e739f55c-0ca9-442e-b6d0-dc8144acafd0-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:03.814262 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.814223    2550 generic.go:356] "Generic (PLEG): container finished" podID="0cc77959-6e75-41db-8767-858a6f6c9052" containerID="5d61df8d4fe609e48a0da4ff5f4e7b87f66a8402694d0e5264bd7373534c3cd0" exitCode=0
Sep 15 19:14:03.814695 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.814279    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"
Sep 15 19:14:03.814695 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.814328    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv" event={"ID":"0cc77959-6e75-41db-8767-858a6f6c9052","Type":"ContainerDied","Data":"5d61df8d4fe609e48a0da4ff5f4e7b87f66a8402694d0e5264bd7373534c3cd0"}
Sep 15 19:14:03.814695 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.814395    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv" event={"ID":"0cc77959-6e75-41db-8767-858a6f6c9052","Type":"ContainerDied","Data":"4ad152980ff32693afec7f6b207f52b8c0164160d2ee276eba9f6e6f7d4d9f2f"}
Sep 15 19:14:03.814695 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.814429    2550 scope.go:117] "RemoveContainer" containerID="5d61df8d4fe609e48a0da4ff5f4e7b87f66a8402694d0e5264bd7373534c3cd0"
Sep 15 19:14:03.815869 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.815827    2550 generic.go:356] "Generic (PLEG): container finished" podID="e739f55c-0ca9-442e-b6d0-dc8144acafd0" containerID="a9db53c8dd3afe9bbded6d21a35afb2d6dd727659a2559777451eaf238b5e717" exitCode=0
Sep 15 19:14:03.815932 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.815878    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"
Sep 15 19:14:03.815932 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.815915    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp" event={"ID":"e739f55c-0ca9-442e-b6d0-dc8144acafd0","Type":"ContainerDied","Data":"a9db53c8dd3afe9bbded6d21a35afb2d6dd727659a2559777451eaf238b5e717"}
Sep 15 19:14:03.816016 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.815966    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp" event={"ID":"e739f55c-0ca9-442e-b6d0-dc8144acafd0","Type":"ContainerDied","Data":"c8d2bb754d8d49e0ebbdbd6221cd55112b9ba22290ca691bb306cb0fdd2a2dab"}
Sep 15 19:14:03.825134 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.825114    2550 scope.go:117] "RemoveContainer" containerID="5d61df8d4fe609e48a0da4ff5f4e7b87f66a8402694d0e5264bd7373534c3cd0"
Sep 15 19:14:03.825472 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:14:03.825446    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d61df8d4fe609e48a0da4ff5f4e7b87f66a8402694d0e5264bd7373534c3cd0\": container with ID starting with 5d61df8d4fe609e48a0da4ff5f4e7b87f66a8402694d0e5264bd7373534c3cd0 not found: ID does not exist" containerID="5d61df8d4fe609e48a0da4ff5f4e7b87f66a8402694d0e5264bd7373534c3cd0"
Sep 15 19:14:03.825583 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.825478    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d61df8d4fe609e48a0da4ff5f4e7b87f66a8402694d0e5264bd7373534c3cd0"} err="failed to get container status \"5d61df8d4fe609e48a0da4ff5f4e7b87f66a8402694d0e5264bd7373534c3cd0\": rpc error: code = NotFound desc = could not find container \"5d61df8d4fe609e48a0da4ff5f4e7b87f66a8402694d0e5264bd7373534c3cd0\": container with ID starting with 5d61df8d4fe609e48a0da4ff5f4e7b87f66a8402694d0e5264bd7373534c3cd0 not found: ID does not exist"
Sep 15 19:14:03.825583 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.825494    2550 scope.go:117] "RemoveContainer" containerID="a9db53c8dd3afe9bbded6d21a35afb2d6dd727659a2559777451eaf238b5e717"
Sep 15 19:14:03.833495 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.833477    2550 scope.go:117] "RemoveContainer" containerID="a9db53c8dd3afe9bbded6d21a35afb2d6dd727659a2559777451eaf238b5e717"
Sep 15 19:14:03.833738 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:14:03.833721    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9db53c8dd3afe9bbded6d21a35afb2d6dd727659a2559777451eaf238b5e717\": container with ID starting with a9db53c8dd3afe9bbded6d21a35afb2d6dd727659a2559777451eaf238b5e717 not found: ID does not exist" containerID="a9db53c8dd3afe9bbded6d21a35afb2d6dd727659a2559777451eaf238b5e717"
Sep 15 19:14:03.833794 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.833744    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9db53c8dd3afe9bbded6d21a35afb2d6dd727659a2559777451eaf238b5e717"} err="failed to get container status \"a9db53c8dd3afe9bbded6d21a35afb2d6dd727659a2559777451eaf238b5e717\": rpc error: code = NotFound desc = could not find container \"a9db53c8dd3afe9bbded6d21a35afb2d6dd727659a2559777451eaf238b5e717\": container with ID starting with a9db53c8dd3afe9bbded6d21a35afb2d6dd727659a2559777451eaf238b5e717 not found: ID does not exist"
Sep 15 19:14:03.839421 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.839399    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"]
Sep 15 19:14:03.842489 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.842466    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-74ef1f-84d7f5d767-l6bgv"]
Sep 15 19:14:03.851429 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.851408    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"]
Sep 15 19:14:03.854640 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:03.854617    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-74ef1f-695df95b6-vp2xp"]
Sep 15 19:14:04.296648 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:04.296614    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cc77959-6e75-41db-8767-858a6f6c9052" path="/var/lib/kubelet/pods/0cc77959-6e75-41db-8767-858a6f6c9052/volumes"
Sep 15 19:14:04.297108 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:04.297087    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e739f55c-0ca9-442e-b6d0-dc8144acafd0" path="/var/lib/kubelet/pods/e739f55c-0ca9-442e-b6d0-dc8144acafd0/volumes"
Sep 15 19:14:07.014475 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.014437    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w" podUID="13718b72-1a1a-4cd5-a0a3-e32e991ca1f0" containerName="main" containerID="cri-o://478b1391b04b99bc8b3ec7067eb8bc48c149d0e086962cbcf16a20c9afa1b98a" gracePeriod=45
Sep 15 19:14:07.020424 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.020397    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z" podUID="626300c6-fa80-4ac5-973c-0546c076bdaa" containerName="main" containerID="cri-o://cea8bb640cf8a451f1d42ffa47a59df93e07651560e8b17b65e40635f685340d" gracePeriod=45
Sep 15 19:14:07.299104 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.299080    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:14:07.302002 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.301973    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:14:07.369943 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.369913    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-dshm\") pod \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") "
Sep 15 19:14:07.370103 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.369962    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-model-cache\") pod \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") "
Sep 15 19:14:07.370103 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.370002    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-home\") pod \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") "
Sep 15 19:14:07.370103 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.370029    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-tls-certs\") pod \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") "
Sep 15 19:14:07.370103 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.370053    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-home\") pod \"626300c6-fa80-4ac5-973c-0546c076bdaa\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") "
Sep 15 19:14:07.370103 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.370099    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-model-cache\") pod \"626300c6-fa80-4ac5-973c-0546c076bdaa\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") "
Sep 15 19:14:07.370402 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.370128    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9bj4\" (UniqueName: \"kubernetes.io/projected/626300c6-fa80-4ac5-973c-0546c076bdaa-kube-api-access-h9bj4\") pod \"626300c6-fa80-4ac5-973c-0546c076bdaa\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") "
Sep 15 19:14:07.370402 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.370160    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-dshm\") pod \"626300c6-fa80-4ac5-973c-0546c076bdaa\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") "
Sep 15 19:14:07.370402 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.370173    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trsvl\" (UniqueName: \"kubernetes.io/projected/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-kube-api-access-trsvl\") pod \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") "
Sep 15 19:14:07.370402 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.370204    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-tmp-dir\") pod \"626300c6-fa80-4ac5-973c-0546c076bdaa\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") "
Sep 15 19:14:07.370402 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.370227    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626300c6-fa80-4ac5-973c-0546c076bdaa-tls-certs\") pod \"626300c6-fa80-4ac5-973c-0546c076bdaa\" (UID: \"626300c6-fa80-4ac5-973c-0546c076bdaa\") "
Sep 15 19:14:07.370402 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.370244    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-tmp-dir\") pod \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\" (UID: \"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0\") "
Sep 15 19:14:07.370402 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.370239    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "13718b72-1a1a-4cd5-a0a3-e32e991ca1f0" (UID: "13718b72-1a1a-4cd5-a0a3-e32e991ca1f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.370751 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.370615    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-model-cache\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.370751 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.370694    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-home" (OuterVolumeSpecName: "home") pod "13718b72-1a1a-4cd5-a0a3-e32e991ca1f0" (UID: "13718b72-1a1a-4cd5-a0a3-e32e991ca1f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.370863 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.370838    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "626300c6-fa80-4ac5-973c-0546c076bdaa" (UID: "626300c6-fa80-4ac5-973c-0546c076bdaa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.371602 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.371566    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-home" (OuterVolumeSpecName: "home") pod "626300c6-fa80-4ac5-973c-0546c076bdaa" (UID: "626300c6-fa80-4ac5-973c-0546c076bdaa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.371879 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.371855    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-model-cache" (OuterVolumeSpecName: "model-cache") pod "626300c6-fa80-4ac5-973c-0546c076bdaa" (UID: "626300c6-fa80-4ac5-973c-0546c076bdaa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.371984 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.371913    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13718b72-1a1a-4cd5-a0a3-e32e991ca1f0" (UID: "13718b72-1a1a-4cd5-a0a3-e32e991ca1f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.373361 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.373015    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13718b72-1a1a-4cd5-a0a3-e32e991ca1f0" (UID: "13718b72-1a1a-4cd5-a0a3-e32e991ca1f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:14:07.373994 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.373970    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-dshm" (OuterVolumeSpecName: "dshm") pod "626300c6-fa80-4ac5-973c-0546c076bdaa" (UID: "626300c6-fa80-4ac5-973c-0546c076bdaa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.374389 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.374360    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626300c6-fa80-4ac5-973c-0546c076bdaa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "626300c6-fa80-4ac5-973c-0546c076bdaa" (UID: "626300c6-fa80-4ac5-973c-0546c076bdaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:14:07.374487 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.374472    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-dshm" (OuterVolumeSpecName: "dshm") pod "13718b72-1a1a-4cd5-a0a3-e32e991ca1f0" (UID: "13718b72-1a1a-4cd5-a0a3-e32e991ca1f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.374625 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.374598    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626300c6-fa80-4ac5-973c-0546c076bdaa-kube-api-access-h9bj4" (OuterVolumeSpecName: "kube-api-access-h9bj4") pod "626300c6-fa80-4ac5-973c-0546c076bdaa" (UID: "626300c6-fa80-4ac5-973c-0546c076bdaa"). InnerVolumeSpecName "kube-api-access-h9bj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:07.374820 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.374801    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-kube-api-access-trsvl" (OuterVolumeSpecName: "kube-api-access-trsvl") pod "13718b72-1a1a-4cd5-a0a3-e32e991ca1f0" (UID: "13718b72-1a1a-4cd5-a0a3-e32e991ca1f0"). InnerVolumeSpecName "kube-api-access-trsvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:07.471351 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.471298    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-tls-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.471351 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.471347    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-home\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.471351 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.471356    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-model-cache\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.471568 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.471368    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9bj4\" (UniqueName: \"kubernetes.io/projected/626300c6-fa80-4ac5-973c-0546c076bdaa-kube-api-access-h9bj4\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.471568 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.471377    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-dshm\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.471568 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.471387    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trsvl\" (UniqueName: \"kubernetes.io/projected/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-kube-api-access-trsvl\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.471568 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.471395    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626300c6-fa80-4ac5-973c-0546c076bdaa-tmp-dir\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.471568 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.471404    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626300c6-fa80-4ac5-973c-0546c076bdaa-tls-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.471568 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.471411    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-tmp-dir\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.471568 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.471418    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-dshm\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.471568 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.471426    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0-home\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.838825 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.838793    2550 generic.go:356] "Generic (PLEG): container finished" podID="626300c6-fa80-4ac5-973c-0546c076bdaa" containerID="cea8bb640cf8a451f1d42ffa47a59df93e07651560e8b17b65e40635f685340d" exitCode=0
Sep 15 19:14:07.838992 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.838835    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"
Sep 15 19:14:07.838992 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.838876    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z" event={"ID":"626300c6-fa80-4ac5-973c-0546c076bdaa","Type":"ContainerDied","Data":"cea8bb640cf8a451f1d42ffa47a59df93e07651560e8b17b65e40635f685340d"}
Sep 15 19:14:07.838992 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.838914    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z" event={"ID":"626300c6-fa80-4ac5-973c-0546c076bdaa","Type":"ContainerDied","Data":"249b78dc63b2e9659a8516c5bb28ce4a1b6546d84e45b56355cbd37c4f9aeff5"}
Sep 15 19:14:07.838992 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.838934    2550 scope.go:117] "RemoveContainer" containerID="cea8bb640cf8a451f1d42ffa47a59df93e07651560e8b17b65e40635f685340d"
Sep 15 19:14:07.840614 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.840591    2550 generic.go:356] "Generic (PLEG): container finished" podID="13718b72-1a1a-4cd5-a0a3-e32e991ca1f0" containerID="478b1391b04b99bc8b3ec7067eb8bc48c149d0e086962cbcf16a20c9afa1b98a" exitCode=0
Sep 15 19:14:07.840741 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.840626    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"
Sep 15 19:14:07.840741 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.840656    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w" event={"ID":"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0","Type":"ContainerDied","Data":"478b1391b04b99bc8b3ec7067eb8bc48c149d0e086962cbcf16a20c9afa1b98a"}
Sep 15 19:14:07.840741 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.840697    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w" event={"ID":"13718b72-1a1a-4cd5-a0a3-e32e991ca1f0","Type":"ContainerDied","Data":"c75d7c7bf8e59e1a90073f513a7d886cc49d5d87d75bb79679af1eed608e37c6"}
Sep 15 19:14:07.851757 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.849478    2550 scope.go:117] "RemoveContainer" containerID="cea8bb640cf8a451f1d42ffa47a59df93e07651560e8b17b65e40635f685340d"
Sep 15 19:14:07.851757 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:14:07.851405    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea8bb640cf8a451f1d42ffa47a59df93e07651560e8b17b65e40635f685340d\": container with ID starting with cea8bb640cf8a451f1d42ffa47a59df93e07651560e8b17b65e40635f685340d not found: ID does not exist" containerID="cea8bb640cf8a451f1d42ffa47a59df93e07651560e8b17b65e40635f685340d"
Sep 15 19:14:07.851757 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.851444    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea8bb640cf8a451f1d42ffa47a59df93e07651560e8b17b65e40635f685340d"} err="failed to get container status \"cea8bb640cf8a451f1d42ffa47a59df93e07651560e8b17b65e40635f685340d\": rpc error: code = NotFound desc = could not find container \"cea8bb640cf8a451f1d42ffa47a59df93e07651560e8b17b65e40635f685340d\": container with ID starting with cea8bb640cf8a451f1d42ffa47a59df93e07651560e8b17b65e40635f685340d not found: ID does not exist"
Sep 15 19:14:07.851757 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.851466    2550 scope.go:117] "RemoveContainer" containerID="478b1391b04b99bc8b3ec7067eb8bc48c149d0e086962cbcf16a20c9afa1b98a"
Sep 15 19:14:07.860861 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.860845    2550 scope.go:117] "RemoveContainer" containerID="478b1391b04b99bc8b3ec7067eb8bc48c149d0e086962cbcf16a20c9afa1b98a"
Sep 15 19:14:07.861112 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:14:07.861090    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"478b1391b04b99bc8b3ec7067eb8bc48c149d0e086962cbcf16a20c9afa1b98a\": container with ID starting with 478b1391b04b99bc8b3ec7067eb8bc48c149d0e086962cbcf16a20c9afa1b98a not found: ID does not exist" containerID="478b1391b04b99bc8b3ec7067eb8bc48c149d0e086962cbcf16a20c9afa1b98a"
Sep 15 19:14:07.861161 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.861119    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"478b1391b04b99bc8b3ec7067eb8bc48c149d0e086962cbcf16a20c9afa1b98a"} err="failed to get container status \"478b1391b04b99bc8b3ec7067eb8bc48c149d0e086962cbcf16a20c9afa1b98a\": rpc error: code = NotFound desc = could not find container \"478b1391b04b99bc8b3ec7067eb8bc48c149d0e086962cbcf16a20c9afa1b98a\": container with ID starting with 478b1391b04b99bc8b3ec7067eb8bc48c149d0e086962cbcf16a20c9afa1b98a not found: ID does not exist"
Sep 15 19:14:07.864946 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.864924    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"]
Sep 15 19:14:07.868627 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.868605    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/premium-w0-74ef1f-kserve-57db786857-nfn5z"]
Sep 15 19:14:07.877407 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.877385    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"]
Sep 15 19:14:07.879049 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:07.879030    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/embedding-w0-74ef1f-kserve-7dd8dd48d5-rqs8w"]
Sep 15 19:14:08.297454 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:08.297424    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13718b72-1a1a-4cd5-a0a3-e32e991ca1f0" path="/var/lib/kubelet/pods/13718b72-1a1a-4cd5-a0a3-e32e991ca1f0/volumes"
Sep 15 19:14:08.297958 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:08.297943    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="626300c6-fa80-4ac5-973c-0546c076bdaa" path="/var/lib/kubelet/pods/626300c6-fa80-4ac5-973c-0546c076bdaa/volumes"
Sep 15 19:14:10.229528 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:10.229496    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"]
Sep 15 19:14:10.229960 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:10.229743    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99" podUID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" containerName="istio-proxy" containerID="cri-o://2889524a350cc2f423bba5ccbd7dfb0ef4492c2e3f1e049cb30559336142cc70" gracePeriod=30
Sep 15 19:14:12.675469 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:12.675434    2550 patch_prober.go:28] interesting pod/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 19:14:12.675844 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:12.675492    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99" podUID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 19:14:15.487196 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.487170    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:14:15.650492 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.650448    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flvgc\" (UniqueName: \"kubernetes.io/projected/ec73c8b1-223d-4859-8a85-d2faeabd20c9-kube-api-access-flvgc\") pod \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") "
Sep 15 19:14:15.650492 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.650499    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-workload-certs\") pod \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") "
Sep 15 19:14:15.650744 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.650531    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-token\") pod \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") "
Sep 15 19:14:15.650744 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.650563    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istiod-ca-cert\") pod \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") "
Sep 15 19:14:15.650744 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.650645    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-data\") pod \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") "
Sep 15 19:14:15.650744 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.650674    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-workload-socket\") pod \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") "
Sep 15 19:14:15.650744 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.650727    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-credential-socket\") pod \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") "
Sep 15 19:14:15.651007 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.650769    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-podinfo\") pod \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") "
Sep 15 19:14:15.651007 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.650799    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-envoy\") pod \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\" (UID: \"ec73c8b1-223d-4859-8a85-d2faeabd20c9\") "
Sep 15 19:14:15.651007 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.650815    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec73c8b1-223d-4859-8a85-d2faeabd20c9" (UID: "ec73c8b1-223d-4859-8a85-d2faeabd20c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:15.651007 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.650955    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec73c8b1-223d-4859-8a85-d2faeabd20c9" (UID: "ec73c8b1-223d-4859-8a85-d2faeabd20c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:15.651224 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.651023    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec73c8b1-223d-4859-8a85-d2faeabd20c9" (UID: "ec73c8b1-223d-4859-8a85-d2faeabd20c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:15.651224 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.651053    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec73c8b1-223d-4859-8a85-d2faeabd20c9" (UID: "ec73c8b1-223d-4859-8a85-d2faeabd20c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:15.651224 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.651198    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-data\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:15.651224 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.651216    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-workload-socket\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:15.651429 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.651231    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-credential-socket\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:15.651429 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.651245    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-workload-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:15.651429 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.651262    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec73c8b1-223d-4859-8a85-d2faeabd20c9" (UID: "ec73c8b1-223d-4859-8a85-d2faeabd20c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:14:15.652764 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.652735    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec73c8b1-223d-4859-8a85-d2faeabd20c9" (UID: "ec73c8b1-223d-4859-8a85-d2faeabd20c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:15.652885 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.652788    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec73c8b1-223d-4859-8a85-d2faeabd20c9-kube-api-access-flvgc" (OuterVolumeSpecName: "kube-api-access-flvgc") pod "ec73c8b1-223d-4859-8a85-d2faeabd20c9" (UID: "ec73c8b1-223d-4859-8a85-d2faeabd20c9"). InnerVolumeSpecName "kube-api-access-flvgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:15.653149 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.653127    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec73c8b1-223d-4859-8a85-d2faeabd20c9" (UID: "ec73c8b1-223d-4859-8a85-d2faeabd20c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:15.653241 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.653166    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec73c8b1-223d-4859-8a85-d2faeabd20c9" (UID: "ec73c8b1-223d-4859-8a85-d2faeabd20c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:14:15.751768 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.751735    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-envoy\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:15.751768 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.751765    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flvgc\" (UniqueName: \"kubernetes.io/projected/ec73c8b1-223d-4859-8a85-d2faeabd20c9-kube-api-access-flvgc\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:15.751768 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.751776    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-token\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:15.752019 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.751785    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istiod-ca-cert\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:15.752019 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.751794    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec73c8b1-223d-4859-8a85-d2faeabd20c9-istio-podinfo\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:14:15.884425 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.884384    2550 generic.go:356] "Generic (PLEG): container finished" podID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" containerID="2889524a350cc2f423bba5ccbd7dfb0ef4492c2e3f1e049cb30559336142cc70" exitCode=0
Sep 15 19:14:15.884583 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.884434    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"
Sep 15 19:14:15.884583 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.884438    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99" event={"ID":"ec73c8b1-223d-4859-8a85-d2faeabd20c9","Type":"ContainerDied","Data":"2889524a350cc2f423bba5ccbd7dfb0ef4492c2e3f1e049cb30559336142cc70"}
Sep 15 19:14:15.884583 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.884491    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99" event={"ID":"ec73c8b1-223d-4859-8a85-d2faeabd20c9","Type":"ContainerDied","Data":"98df13e04a129551f5229b1a0c23a3497efe0f82da46316f40bf840e6c04c5a3"}
Sep 15 19:14:15.884583 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.884521    2550 scope.go:117] "RemoveContainer" containerID="2889524a350cc2f423bba5ccbd7dfb0ef4492c2e3f1e049cb30559336142cc70"
Sep 15 19:14:15.894601 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.894580    2550 scope.go:117] "RemoveContainer" containerID="7aacf60a3dee30353f3fe29939cfdfbb253b997a13abe797c9d2dceaef9a9ec4"
Sep 15 19:14:15.902452 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.902433    2550 scope.go:117] "RemoveContainer" containerID="2889524a350cc2f423bba5ccbd7dfb0ef4492c2e3f1e049cb30559336142cc70"
Sep 15 19:14:15.902739 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:14:15.902719    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2889524a350cc2f423bba5ccbd7dfb0ef4492c2e3f1e049cb30559336142cc70\": container with ID starting with 2889524a350cc2f423bba5ccbd7dfb0ef4492c2e3f1e049cb30559336142cc70 not found: ID does not exist" containerID="2889524a350cc2f423bba5ccbd7dfb0ef4492c2e3f1e049cb30559336142cc70"
Sep 15 19:14:15.902785 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.902747    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2889524a350cc2f423bba5ccbd7dfb0ef4492c2e3f1e049cb30559336142cc70"} err="failed to get container status \"2889524a350cc2f423bba5ccbd7dfb0ef4492c2e3f1e049cb30559336142cc70\": rpc error: code = NotFound desc = could not find container \"2889524a350cc2f423bba5ccbd7dfb0ef4492c2e3f1e049cb30559336142cc70\": container with ID starting with 2889524a350cc2f423bba5ccbd7dfb0ef4492c2e3f1e049cb30559336142cc70 not found: ID does not exist"
Sep 15 19:14:15.902785 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.902764    2550 scope.go:117] "RemoveContainer" containerID="7aacf60a3dee30353f3fe29939cfdfbb253b997a13abe797c9d2dceaef9a9ec4"
Sep 15 19:14:15.902968 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:14:15.902949    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aacf60a3dee30353f3fe29939cfdfbb253b997a13abe797c9d2dceaef9a9ec4\": container with ID starting with 7aacf60a3dee30353f3fe29939cfdfbb253b997a13abe797c9d2dceaef9a9ec4 not found: ID does not exist" containerID="7aacf60a3dee30353f3fe29939cfdfbb253b997a13abe797c9d2dceaef9a9ec4"
Sep 15 19:14:15.903006 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.902973    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aacf60a3dee30353f3fe29939cfdfbb253b997a13abe797c9d2dceaef9a9ec4"} err="failed to get container status \"7aacf60a3dee30353f3fe29939cfdfbb253b997a13abe797c9d2dceaef9a9ec4\": rpc error: code = NotFound desc = could not find container \"7aacf60a3dee30353f3fe29939cfdfbb253b997a13abe797c9d2dceaef9a9ec4\": container with ID starting with 7aacf60a3dee30353f3fe29939cfdfbb253b997a13abe797c9d2dceaef9a9ec4 not found: ID does not exist"
Sep 15 19:14:15.908067 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.908043    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"]
Sep 15 19:14:15.912161 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:15.912138    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-74ef1f-openshift-default-59774bc458-pfk99"]
Sep 15 19:14:16.295655 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:16.295624    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" path="/var/lib/kubelet/pods/ec73c8b1-223d-4859-8a85-d2faeabd20c9/volumes"
Sep 15 19:14:17.336163 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:17.336123    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:14:33.310722 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:33.310645    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:14:46.207110 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:46.207072    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:14:59.502304 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:14:59.502265    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"]
Sep 15 19:15:00.148280 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148224    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7073b8-29824995-jvfc2"]
Sep 15 19:15:00.148733 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148717    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" containerName="istio-proxy"
Sep 15 19:15:00.148826 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148742    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" containerName="istio-proxy"
Sep 15 19:15:00.148826 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148760    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626300c6-fa80-4ac5-973c-0546c076bdaa" containerName="main"
Sep 15 19:15:00.148826 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148769    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="626300c6-fa80-4ac5-973c-0546c076bdaa" containerName="main"
Sep 15 19:15:00.148826 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148802    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9433fdc9-33b2-4ab4-a148-59e559692414" containerName="payload-processing"
Sep 15 19:15:00.148826 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148812    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9433fdc9-33b2-4ab4-a148-59e559692414" containerName="payload-processing"
Sep 15 19:15:00.149036 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148830    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cc77959-6e75-41db-8767-858a6f6c9052" containerName="maas-api"
Sep 15 19:15:00.149036 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148839    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cc77959-6e75-41db-8767-858a6f6c9052" containerName="maas-api"
Sep 15 19:15:00.149036 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148852    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13718b72-1a1a-4cd5-a0a3-e32e991ca1f0" containerName="main"
Sep 15 19:15:00.149036 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148861    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="13718b72-1a1a-4cd5-a0a3-e32e991ca1f0" containerName="main"
Sep 15 19:15:00.149036 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148877    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4e0a64f-eeff-41bb-ad36-b0fee8a6896c" containerName="payload-pre-processing"
Sep 15 19:15:00.149036 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148885    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4e0a64f-eeff-41bb-ad36-b0fee8a6896c" containerName="payload-pre-processing"
Sep 15 19:15:00.149036 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148899    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e739f55c-0ca9-442e-b6d0-dc8144acafd0" containerName="payload-pre-processing"
Sep 15 19:15:00.149036 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148908    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e739f55c-0ca9-442e-b6d0-dc8144acafd0" containerName="payload-pre-processing"
Sep 15 19:15:00.149036 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148983    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" containerName="istio-proxy"
Sep 15 19:15:00.149036 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148991    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4e0a64f-eeff-41bb-ad36-b0fee8a6896c" containerName="payload-pre-processing"
Sep 15 19:15:00.149036 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.148998    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="626300c6-fa80-4ac5-973c-0546c076bdaa" containerName="main"
Sep 15 19:15:00.149036 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.149004    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9433fdc9-33b2-4ab4-a148-59e559692414" containerName="payload-processing"
Sep 15 19:15:00.149036 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.149012    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cc77959-6e75-41db-8767-858a6f6c9052" containerName="maas-api"
Sep 15 19:15:00.149036 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.149019    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e739f55c-0ca9-442e-b6d0-dc8144acafd0" containerName="payload-pre-processing"
Sep 15 19:15:00.149036 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.149028    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="13718b72-1a1a-4cd5-a0a3-e32e991ca1f0" containerName="main"
Sep 15 19:15:00.153613 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.153586    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824995-rrcv4"]
Sep 15 19:15:00.153726 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.153701    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7073b8-29824995-jvfc2"
Sep 15 19:15:00.154220 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.154195    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" containerName="istio-proxy"
Sep 15 19:15:00.154278 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.154226    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" containerName="istio-proxy"
Sep 15 19:15:00.154354 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.154341    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec73c8b1-223d-4859-8a85-d2faeabd20c9" containerName="istio-proxy"
Sep 15 19:15:00.156865 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.156846    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7xzj\""
Sep 15 19:15:00.157584 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.157565    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-rrcv4"
Sep 15 19:15:00.165282 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.165258    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7073b8-29824995-jvfc2"]
Sep 15 19:15:00.166685 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.166666    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7073b8-29824995-jvfc2"
Sep 15 19:15:00.172210 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.171027    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824995-rrcv4"]
Sep 15 19:15:00.176175 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.175993    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-rrcv4"
Sep 15 19:15:00.301943 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.301915    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:15:00.451635 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.451613    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7073b8-29824995-jvfc2"]
Sep 15 19:15:00.454062 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:15:00.454034    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode286ffc5_bd97_4023_8482_1aadca768ae6.slice/crio-e9eddd64db98179cab9a54d95066e90b04591d058227662e1bd29b2fc7dc2f96 WatchSource:0}: Error finding container e9eddd64db98179cab9a54d95066e90b04591d058227662e1bd29b2fc7dc2f96: Status 404 returned error can't find the container with id e9eddd64db98179cab9a54d95066e90b04591d058227662e1bd29b2fc7dc2f96
Sep 15 19:15:00.460641 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:00.460614    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824995-rrcv4"]
Sep 15 19:15:00.462186 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:15:00.462164    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671eb34d_80cc_42fb_b143_14d563deadda.slice/crio-90ef6ace823813719f2718aa6e87d8e466d623ae5caeb7d0898ea0f2840d17b9 WatchSource:0}: Error finding container 90ef6ace823813719f2718aa6e87d8e466d623ae5caeb7d0898ea0f2840d17b9: Status 404 returned error can't find the container with id 90ef6ace823813719f2718aa6e87d8e466d623ae5caeb7d0898ea0f2840d17b9
Sep 15 19:15:01.100201 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:01.100164    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7073b8-29824995-jvfc2" event={"ID":"e286ffc5-bd97-4023-8482-1aadca768ae6","Type":"ContainerStarted","Data":"e9eddd64db98179cab9a54d95066e90b04591d058227662e1bd29b2fc7dc2f96"}
Sep 15 19:15:01.101351 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:01.101306    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-rrcv4" event={"ID":"671eb34d-80cc-42fb-b143-14d563deadda","Type":"ContainerStarted","Data":"90ef6ace823813719f2718aa6e87d8e466d623ae5caeb7d0898ea0f2840d17b9"}
Sep 15 19:15:03.113349 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:03.113292    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-rrcv4" event={"ID":"671eb34d-80cc-42fb-b143-14d563deadda","Type":"ContainerStarted","Data":"191dec4e8dff4b01685d66113efb5397568915625e590ba2893bab0c640727f3"}
Sep 15 19:15:03.114799 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:03.114776    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7073b8-29824995-jvfc2" event={"ID":"e286ffc5-bd97-4023-8482-1aadca768ae6","Type":"ContainerStarted","Data":"fd4c45d46bb0a6b439192688ab7e6e5b4bf5dadb896964e83509c345ed04cacd"}
Sep 15 19:15:03.128285 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:03.128238    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-rrcv4" podStartSLOduration=0.995027558 podStartE2EDuration="3.1282253s" totalImagesPullingTime="2.133197742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:03.126627007 +0000 UTC m=+1465.403395867" watchObservedRunningTime="2026-09-15 19:15:03.1282253 +0000 UTC m=+1465.404994202"
Sep 15 19:15:03.141654 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:03.141584    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7073b8-29824995-jvfc2" podStartSLOduration=0.995642722 podStartE2EDuration="3.141570439s" totalImagesPullingTime="2.145927717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:03.141159035 +0000 UTC m=+1465.417927891" watchObservedRunningTime="2026-09-15 19:15:03.141570439 +0000 UTC m=+1465.418339295"
Sep 15 19:15:04.120952 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:04.120916    2550 generic.go:356] "Generic (PLEG): container finished" podID="e286ffc5-bd97-4023-8482-1aadca768ae6" containerID="fd4c45d46bb0a6b439192688ab7e6e5b4bf5dadb896964e83509c345ed04cacd" exitCode=0
Sep 15 19:15:04.121423 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:04.120966    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7073b8-29824995-jvfc2" event={"ID":"e286ffc5-bd97-4023-8482-1aadca768ae6","Type":"ContainerDied","Data":"fd4c45d46bb0a6b439192688ab7e6e5b4bf5dadb896964e83509c345ed04cacd"}
Sep 15 19:15:04.122581 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:04.122559    2550 generic.go:356] "Generic (PLEG): container finished" podID="671eb34d-80cc-42fb-b143-14d563deadda" containerID="191dec4e8dff4b01685d66113efb5397568915625e590ba2893bab0c640727f3" exitCode=0
Sep 15 19:15:04.122696 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:04.122604    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-rrcv4" event={"ID":"671eb34d-80cc-42fb-b143-14d563deadda","Type":"ContainerDied","Data":"191dec4e8dff4b01685d66113efb5397568915625e590ba2893bab0c640727f3"}
Sep 15 19:15:05.274658 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:05.274628    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7073b8-29824995-jvfc2"
Sep 15 19:15:05.297005 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:05.296980    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-rrcv4"
Sep 15 19:15:06.135242 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:06.135208    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7073b8-29824995-jvfc2"
Sep 15 19:15:06.135453 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:06.135248    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7073b8-29824995-jvfc2" event={"ID":"e286ffc5-bd97-4023-8482-1aadca768ae6","Type":"ContainerDied","Data":"e9eddd64db98179cab9a54d95066e90b04591d058227662e1bd29b2fc7dc2f96"}
Sep 15 19:15:06.135453 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:06.135295    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9eddd64db98179cab9a54d95066e90b04591d058227662e1bd29b2fc7dc2f96"
Sep 15 19:15:06.136650 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:06.136619    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-rrcv4"
Sep 15 19:15:06.136788 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:06.136644    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-rrcv4" event={"ID":"671eb34d-80cc-42fb-b143-14d563deadda","Type":"ContainerDied","Data":"90ef6ace823813719f2718aa6e87d8e466d623ae5caeb7d0898ea0f2840d17b9"}
Sep 15 19:15:06.136788 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:06.136690    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90ef6ace823813719f2718aa6e87d8e466d623ae5caeb7d0898ea0f2840d17b9"
Sep 15 19:15:10.101965 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.101924    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:15:10.328467 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.328437    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"]
Sep 15 19:15:10.328874 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.328858    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="671eb34d-80cc-42fb-b143-14d563deadda" containerName="cleanup"
Sep 15 19:15:10.328931 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.328878    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="671eb34d-80cc-42fb-b143-14d563deadda" containerName="cleanup"
Sep 15 19:15:10.328931 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.328891    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e286ffc5-bd97-4023-8482-1aadca768ae6" containerName="cleanup"
Sep 15 19:15:10.328931 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.328897    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e286ffc5-bd97-4023-8482-1aadca768ae6" containerName="cleanup"
Sep 15 19:15:10.329044 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.328960    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="671eb34d-80cc-42fb-b143-14d563deadda" containerName="cleanup"
Sep 15 19:15:10.329044 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.328971    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e286ffc5-bd97-4023-8482-1aadca768ae6" containerName="cleanup"
Sep 15 19:15:10.332035 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.332015    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.335253 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.335225    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-910e11-openshift-default-dockercfg-zsjrc\""
Sep 15 19:15:10.346127 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.346107    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"]
Sep 15 19:15:10.351809 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.351780    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a42c8b-5966-4638-9799-60e7e99afe43-istio-podinfo\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.351922 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.351855    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-workload-certs\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.351922 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.351889    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a42c8b-5966-4638-9799-60e7e99afe43-istio-token\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.352072 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.351921    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44kfx\" (UniqueName: \"kubernetes.io/projected/88a42c8b-5966-4638-9799-60e7e99afe43-kube-api-access-44kfx\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.352072 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.351971    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-istio-envoy\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.352072 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.352009    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a42c8b-5966-4638-9799-60e7e99afe43-istiod-ca-cert\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.352072 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.352036    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-istio-data\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.352072 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.352060    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-credential-socket\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.352251 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.352128    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-workload-socket\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.452705 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.452669    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a42c8b-5966-4638-9799-60e7e99afe43-istio-podinfo\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.452848 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.452747    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-workload-certs\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.452848 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.452780    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a42c8b-5966-4638-9799-60e7e99afe43-istio-token\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.452848 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.452814    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44kfx\" (UniqueName: \"kubernetes.io/projected/88a42c8b-5966-4638-9799-60e7e99afe43-kube-api-access-44kfx\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.453030 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.452854    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-istio-envoy\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.453030 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.452893    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a42c8b-5966-4638-9799-60e7e99afe43-istiod-ca-cert\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.453030 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.452919    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-istio-data\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.453030 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.452942    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-credential-socket\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.453030 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.452993    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-workload-socket\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.453376 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.453345    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-istio-data\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.453453 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.453405    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-workload-certs\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.453524 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.453503    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-credential-socket\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.453597 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.453560    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-workload-socket\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.453752 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.453734    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a42c8b-5966-4638-9799-60e7e99afe43-istiod-ca-cert\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.455272 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.455251    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-istio-envoy\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.455653 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.455634    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a42c8b-5966-4638-9799-60e7e99afe43-istio-podinfo\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.460041 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.460014    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a42c8b-5966-4638-9799-60e7e99afe43-istio-token\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.460345 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.460304    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44kfx\" (UniqueName: \"kubernetes.io/projected/88a42c8b-5966-4638-9799-60e7e99afe43-kube-api-access-44kfx\") pod \"e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") " pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.643770 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.643698    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:10.925188 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.925157    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"]
Sep 15 19:15:10.926624 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:15:10.926599    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a42c8b_5966_4638_9799_60e7e99afe43.slice/crio-e4a5df7461cc69b789190a9491e3e9f2e85db7d2fd3258569c5254ef6c0bc005 WatchSource:0}: Error finding container e4a5df7461cc69b789190a9491e3e9f2e85db7d2fd3258569c5254ef6c0bc005: Status 404 returned error can't find the container with id e4a5df7461cc69b789190a9491e3e9f2e85db7d2fd3258569c5254ef6c0bc005
Sep 15 19:15:10.929153 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.929121    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:15:10.929233 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.929184    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:15:10.929233 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:10.929211    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:15:11.163375 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:11.163337    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t" event={"ID":"88a42c8b-5966-4638-9799-60e7e99afe43","Type":"ContainerStarted","Data":"e32ff4c54f44db03190b8363c40296c6077dfc59dc5b261bd78cb7f204449735"}
Sep 15 19:15:11.163752 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:11.163411    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t" event={"ID":"88a42c8b-5966-4638-9799-60e7e99afe43","Type":"ContainerStarted","Data":"e4a5df7461cc69b789190a9491e3e9f2e85db7d2fd3258569c5254ef6c0bc005"}
Sep 15 19:15:11.181569 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:11.181458    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t" podStartSLOduration=1.181437655 podStartE2EDuration="1.181437655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:11.180303309 +0000 UTC m=+1473.457072168" watchObservedRunningTime="2026-09-15 19:15:11.181437655 +0000 UTC m=+1473.458206509"
Sep 15 19:15:11.644044 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:11.644011    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:11.649090 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:11.649063    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:12.168372 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:12.168344    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:12.169218 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:12.169200    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:14.521731 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.521676    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq" podUID="ab086655-be1e-406f-9d89-bc94b0c2aa56" containerName="main" containerID="cri-o://ba695e462fadadd202d8054b34a9e11a5a71dc366cadf63808517a46ed21c904" gracePeriod=45
Sep 15 19:15:14.775627 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.775575    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:15:14.793887 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.793861    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab086655-be1e-406f-9d89-bc94b0c2aa56-tls-certs\") pod \"ab086655-be1e-406f-9d89-bc94b0c2aa56\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") "
Sep 15 19:15:14.794012 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.793901    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-home\") pod \"ab086655-be1e-406f-9d89-bc94b0c2aa56\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") "
Sep 15 19:15:14.794130 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.794104    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqdfd\" (UniqueName: \"kubernetes.io/projected/ab086655-be1e-406f-9d89-bc94b0c2aa56-kube-api-access-nqdfd\") pod \"ab086655-be1e-406f-9d89-bc94b0c2aa56\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") "
Sep 15 19:15:14.794196 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.794157    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-dshm\") pod \"ab086655-be1e-406f-9d89-bc94b0c2aa56\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") "
Sep 15 19:15:14.794196 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.794170    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-home" (OuterVolumeSpecName: "home") pod "ab086655-be1e-406f-9d89-bc94b0c2aa56" (UID: "ab086655-be1e-406f-9d89-bc94b0c2aa56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:14.794305 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.794249    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-model-cache\") pod \"ab086655-be1e-406f-9d89-bc94b0c2aa56\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") "
Sep 15 19:15:14.794305 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.794289    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-tmp-dir\") pod \"ab086655-be1e-406f-9d89-bc94b0c2aa56\" (UID: \"ab086655-be1e-406f-9d89-bc94b0c2aa56\") "
Sep 15 19:15:14.794781 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.794670    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab086655-be1e-406f-9d89-bc94b0c2aa56" (UID: "ab086655-be1e-406f-9d89-bc94b0c2aa56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:14.794781 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.794750    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab086655-be1e-406f-9d89-bc94b0c2aa56" (UID: "ab086655-be1e-406f-9d89-bc94b0c2aa56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:14.794781 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.794762    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-home\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:15:14.794781 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.794780    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-model-cache\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:15:14.796610 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.796560    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab086655-be1e-406f-9d89-bc94b0c2aa56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab086655-be1e-406f-9d89-bc94b0c2aa56" (UID: "ab086655-be1e-406f-9d89-bc94b0c2aa56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:15:14.796938 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.796917    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-dshm" (OuterVolumeSpecName: "dshm") pod "ab086655-be1e-406f-9d89-bc94b0c2aa56" (UID: "ab086655-be1e-406f-9d89-bc94b0c2aa56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:14.797021 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.796943    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab086655-be1e-406f-9d89-bc94b0c2aa56-kube-api-access-nqdfd" (OuterVolumeSpecName: "kube-api-access-nqdfd") pod "ab086655-be1e-406f-9d89-bc94b0c2aa56" (UID: "ab086655-be1e-406f-9d89-bc94b0c2aa56"). InnerVolumeSpecName "kube-api-access-nqdfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:15:14.895259 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.895229    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqdfd\" (UniqueName: \"kubernetes.io/projected/ab086655-be1e-406f-9d89-bc94b0c2aa56-kube-api-access-nqdfd\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:15:14.895259 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.895259    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-dshm\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:15:14.895471 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.895273    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab086655-be1e-406f-9d89-bc94b0c2aa56-tmp-dir\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:15:14.895471 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:14.895286    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab086655-be1e-406f-9d89-bc94b0c2aa56-tls-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:15:15.191144 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:15.191053    2550 generic.go:356] "Generic (PLEG): container finished" podID="ab086655-be1e-406f-9d89-bc94b0c2aa56" containerID="ba695e462fadadd202d8054b34a9e11a5a71dc366cadf63808517a46ed21c904" exitCode=0
Sep 15 19:15:15.191144 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:15.191097    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"
Sep 15 19:15:15.191144 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:15.191100    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq" event={"ID":"ab086655-be1e-406f-9d89-bc94b0c2aa56","Type":"ContainerDied","Data":"ba695e462fadadd202d8054b34a9e11a5a71dc366cadf63808517a46ed21c904"}
Sep 15 19:15:15.191442 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:15.191150    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq" event={"ID":"ab086655-be1e-406f-9d89-bc94b0c2aa56","Type":"ContainerDied","Data":"85cd2cab41819f2790f2c51a8e7d85747d0fcd20d7f800ea8f90c4a9f56f0a08"}
Sep 15 19:15:15.191442 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:15.191167    2550 scope.go:117] "RemoveContainer" containerID="ba695e462fadadd202d8054b34a9e11a5a71dc366cadf63808517a46ed21c904"
Sep 15 19:15:15.200665 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:15.200653    2550 scope.go:117] "RemoveContainer" containerID="ba695e462fadadd202d8054b34a9e11a5a71dc366cadf63808517a46ed21c904"
Sep 15 19:15:15.200954 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:15:15.200930    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba695e462fadadd202d8054b34a9e11a5a71dc366cadf63808517a46ed21c904\": container with ID starting with ba695e462fadadd202d8054b34a9e11a5a71dc366cadf63808517a46ed21c904 not found: ID does not exist" containerID="ba695e462fadadd202d8054b34a9e11a5a71dc366cadf63808517a46ed21c904"
Sep 15 19:15:15.201044 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:15.200958    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba695e462fadadd202d8054b34a9e11a5a71dc366cadf63808517a46ed21c904"} err="failed to get container status \"ba695e462fadadd202d8054b34a9e11a5a71dc366cadf63808517a46ed21c904\": rpc error: code = NotFound desc = could not find container \"ba695e462fadadd202d8054b34a9e11a5a71dc366cadf63808517a46ed21c904\": container with ID starting with ba695e462fadadd202d8054b34a9e11a5a71dc366cadf63808517a46ed21c904 not found: ID does not exist"
Sep 15 19:15:15.211264 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:15.211238    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"]
Sep 15 19:15:15.215661 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:15.215634    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/auth-test-model-7073b8-kserve-64fc77fcf9-lhdmq"]
Sep 15 19:15:16.295628 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.295593    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab086655-be1e-406f-9d89-bc94b0c2aa56" path="/var/lib/kubelet/pods/ab086655-be1e-406f-9d89-bc94b0c2aa56/volumes"
Sep 15 19:15:16.806240 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.806207    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"]
Sep 15 19:15:16.806606 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.806594    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab086655-be1e-406f-9d89-bc94b0c2aa56" containerName="main"
Sep 15 19:15:16.806662 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.806608    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab086655-be1e-406f-9d89-bc94b0c2aa56" containerName="main"
Sep 15 19:15:16.806698 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.806671    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab086655-be1e-406f-9d89-bc94b0c2aa56" containerName="main"
Sep 15 19:15:16.811151 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.811135    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"
Sep 15 19:15:16.817879 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.817604    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-910e11-dockercfg-2wg6f\""
Sep 15 19:15:16.817958 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.817928    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-910e11\""
Sep 15 19:15:16.829633 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.829605    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"]
Sep 15 19:15:16.889938 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.889899    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"]
Sep 15 19:15:16.893245 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.893224    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"
Sep 15 19:15:16.903241 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.903218    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"]
Sep 15 19:15:16.913508 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.913472    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3103733-cfb4-4438-b0f9-0b6d4819de3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7\" (UID: \"f3103733-cfb4-4438-b0f9-0b6d4819de3b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"
Sep 15 19:15:16.913618 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.913520    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/360935f8-6868-4baf-8ca2-ac769dd652c1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-910e11-996975df7-287b8\" (UID: \"360935f8-6868-4baf-8ca2-ac769dd652c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"
Sep 15 19:15:16.913684 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.913641    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqhs4\" (UniqueName: \"kubernetes.io/projected/360935f8-6868-4baf-8ca2-ac769dd652c1-kube-api-access-dqhs4\") pod \"payload-processing-e2e-api-a-910e11-996975df7-287b8\" (UID: \"360935f8-6868-4baf-8ca2-ac769dd652c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"
Sep 15 19:15:16.913745 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:16.913733    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt7v7\" (UniqueName: \"kubernetes.io/projected/f3103733-cfb4-4438-b0f9-0b6d4819de3b-kube-api-access-mt7v7\") pod \"payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7\" (UID: \"f3103733-cfb4-4438-b0f9-0b6d4819de3b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"
Sep 15 19:15:17.014648 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:17.014616    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqhs4\" (UniqueName: \"kubernetes.io/projected/360935f8-6868-4baf-8ca2-ac769dd652c1-kube-api-access-dqhs4\") pod \"payload-processing-e2e-api-a-910e11-996975df7-287b8\" (UID: \"360935f8-6868-4baf-8ca2-ac769dd652c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"
Sep 15 19:15:17.014798 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:17.014661    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt7v7\" (UniqueName: \"kubernetes.io/projected/f3103733-cfb4-4438-b0f9-0b6d4819de3b-kube-api-access-mt7v7\") pod \"payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7\" (UID: \"f3103733-cfb4-4438-b0f9-0b6d4819de3b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"
Sep 15 19:15:17.014798 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:17.014704    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3103733-cfb4-4438-b0f9-0b6d4819de3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7\" (UID: \"f3103733-cfb4-4438-b0f9-0b6d4819de3b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"
Sep 15 19:15:17.014798 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:17.014721    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/360935f8-6868-4baf-8ca2-ac769dd652c1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-910e11-996975df7-287b8\" (UID: \"360935f8-6868-4baf-8ca2-ac769dd652c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"
Sep 15 19:15:17.015406 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:17.015387    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/360935f8-6868-4baf-8ca2-ac769dd652c1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-910e11-996975df7-287b8\" (UID: \"360935f8-6868-4baf-8ca2-ac769dd652c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"
Sep 15 19:15:17.015686 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:17.015663    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3103733-cfb4-4438-b0f9-0b6d4819de3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7\" (UID: \"f3103733-cfb4-4438-b0f9-0b6d4819de3b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"
Sep 15 19:15:17.023070 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:17.023045    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqhs4\" (UniqueName: \"kubernetes.io/projected/360935f8-6868-4baf-8ca2-ac769dd652c1-kube-api-access-dqhs4\") pod \"payload-processing-e2e-api-a-910e11-996975df7-287b8\" (UID: \"360935f8-6868-4baf-8ca2-ac769dd652c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"
Sep 15 19:15:17.023177 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:17.023159    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt7v7\" (UniqueName: \"kubernetes.io/projected/f3103733-cfb4-4438-b0f9-0b6d4819de3b-kube-api-access-mt7v7\") pod \"payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7\" (UID: \"f3103733-cfb4-4438-b0f9-0b6d4819de3b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"
Sep 15 19:15:17.121525 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:17.121439    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"
Sep 15 19:15:17.208952 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:17.204927    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"
Sep 15 19:15:17.390876 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:17.390852    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"]
Sep 15 19:15:17.393096 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:15:17.393068    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3103733_cfb4_4438_b0f9_0b6d4819de3b.slice/crio-fb8967db9418774a48738ddcdeb5adbb1f65b08920b664abb1eab31dc108c248 WatchSource:0}: Error finding container fb8967db9418774a48738ddcdeb5adbb1f65b08920b664abb1eab31dc108c248: Status 404 returned error can't find the container with id fb8967db9418774a48738ddcdeb5adbb1f65b08920b664abb1eab31dc108c248
Sep 15 19:15:17.483345 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:17.483304    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"]
Sep 15 19:15:17.486157 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:15:17.486134    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360935f8_6868_4baf_8ca2_ac769dd652c1.slice/crio-28ac614b58cf3ba4dee7024f8ecc3a035e4d85bcaa948f965f7d3952f92bd300 WatchSource:0}: Error finding container 28ac614b58cf3ba4dee7024f8ecc3a035e4d85bcaa948f965f7d3952f92bd300: Status 404 returned error can't find the container with id 28ac614b58cf3ba4dee7024f8ecc3a035e4d85bcaa948f965f7d3952f92bd300
Sep 15 19:15:18.218417 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:18.218379    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7" event={"ID":"f3103733-cfb4-4438-b0f9-0b6d4819de3b","Type":"ContainerStarted","Data":"12f2055cd2a80013c9e4bda1146f909e830dec49ddb9a9f575c90887a3a79086"}
Sep 15 19:15:18.218609 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:18.218478    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7" event={"ID":"f3103733-cfb4-4438-b0f9-0b6d4819de3b","Type":"ContainerStarted","Data":"fb8967db9418774a48738ddcdeb5adbb1f65b08920b664abb1eab31dc108c248"}
Sep 15 19:15:18.218609 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:18.218514    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"
Sep 15 19:15:18.220089 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:18.220057    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8" event={"ID":"360935f8-6868-4baf-8ca2-ac769dd652c1","Type":"ContainerStarted","Data":"19f4adce975f677e91df4f2ae5232b241a2b24eaef6075a7f0d737b46524f415"}
Sep 15 19:15:18.220215 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:18.220113    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8" event={"ID":"360935f8-6868-4baf-8ca2-ac769dd652c1","Type":"ContainerStarted","Data":"28ac614b58cf3ba4dee7024f8ecc3a035e4d85bcaa948f965f7d3952f92bd300"}
Sep 15 19:15:18.220215 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:18.220155    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"
Sep 15 19:15:18.234852 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:18.234805    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7" podStartSLOduration=2.234790247 podStartE2EDuration="2.234790247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:18.232929422 +0000 UTC m=+1480.509698279" watchObservedRunningTime="2026-09-15 19:15:18.234790247 +0000 UTC m=+1480.511559103"
Sep 15 19:15:18.250649 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:18.250600    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8" podStartSLOduration=2.250583629 podStartE2EDuration="2.250583629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:18.248211352 +0000 UTC m=+1480.524980203" watchObservedRunningTime="2026-09-15 19:15:18.250583629 +0000 UTC m=+1480.527352518"
Sep 15 19:15:21.115504 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:21.115471    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:15:29.227666 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:29.227633    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"
Sep 15 19:15:29.228164 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:29.227814    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"
Sep 15 19:15:31.301625 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:31.301596    2550 generic.go:356] "Generic (PLEG): container finished" podID="88a42c8b-5966-4638-9799-60e7e99afe43" containerID="e32ff4c54f44db03190b8363c40296c6077dfc59dc5b261bd78cb7f204449735" exitCode=0
Sep 15 19:15:31.302013 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:31.301665    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t" event={"ID":"88a42c8b-5966-4638-9799-60e7e99afe43","Type":"ContainerDied","Data":"e32ff4c54f44db03190b8363c40296c6077dfc59dc5b261bd78cb7f204449735"}
Sep 15 19:15:31.302097 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:31.302079    2550 scope.go:117] "RemoveContainer" containerID="e32ff4c54f44db03190b8363c40296c6077dfc59dc5b261bd78cb7f204449735"
Sep 15 19:15:31.303137 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:31.303112    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:15:31.303183 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:31.303174    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:15:31.303218 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:31.303204    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:15:32.311391 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:32.311352    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t" event={"ID":"88a42c8b-5966-4638-9799-60e7e99afe43","Type":"ContainerStarted","Data":"b22907354f916382ee50416fb30d19bebb38a0b83e2234cf9b54d4555220eb65"}
Sep 15 19:15:32.644150 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:32.644056    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:32.644394 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:32.644368    2550 patch_prober.go:28] interesting pod/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 15 19:15:32.644494 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:32.644438    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t" podUID="88a42c8b-5966-4638-9799-60e7e99afe43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 15 19:15:33.644574 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:33.644536    2550 patch_prober.go:28] interesting pod/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 15 19:15:33.644962 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:33.644590    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t" podUID="88a42c8b-5966-4638-9799-60e7e99afe43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 15 19:15:34.644652 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:34.644615    2550 patch_prober.go:28] interesting pod/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 15 19:15:34.645023 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:34.644669    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t" podUID="88a42c8b-5966-4638-9799-60e7e99afe43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 15 19:15:35.644787 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:35.644748    2550 patch_prober.go:28] interesting pod/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 15 19:15:35.645167 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:35.644806    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t" podUID="88a42c8b-5966-4638-9799-60e7e99afe43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 15 19:15:36.645076 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:36.645037    2550 patch_prober.go:28] interesting pod/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 15 19:15:36.645475 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:36.645094    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t" podUID="88a42c8b-5966-4638-9799-60e7e99afe43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 15 19:15:37.789148 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:37.789115    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:37.789652 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:37.789533    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:37.790681 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:37.790653    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:15:46.930735 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:46.930695    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"]
Sep 15 19:15:46.936894 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:46.936862    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"
Sep 15 19:15:46.939606 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:46.939213    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0a7ab7\""
Sep 15 19:15:46.939606 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:46.939426    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0a7ab7-dockercfg-dkhvr\""
Sep 15 19:15:46.941743 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:46.941721    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"]
Sep 15 19:15:46.956723 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:46.956701    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"]
Sep 15 19:15:46.960220 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:46.960201    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"
Sep 15 19:15:46.975360 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:46.975340    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"]
Sep 15 19:15:47.025916 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.025885    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f86b20-ed93-45f2-bb0b-b94af151517c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5\" (UID: \"74f86b20-ed93-45f2-bb0b-b94af151517c\") " pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"
Sep 15 19:15:47.026103 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.025943    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn5tj\" (UniqueName: \"kubernetes.io/projected/74f86b20-ed93-45f2-bb0b-b94af151517c-kube-api-access-sn5tj\") pod \"payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5\" (UID: \"74f86b20-ed93-45f2-bb0b-b94af151517c\") " pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"
Sep 15 19:15:47.026103 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.026028    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbfx8\" (UniqueName: \"kubernetes.io/projected/d4ea683d-ebb5-4a36-9d07-bf1a26672dec-kube-api-access-nbfx8\") pod \"payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr\" (UID: \"d4ea683d-ebb5-4a36-9d07-bf1a26672dec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"
Sep 15 19:15:47.026103 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.026066    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ea683d-ebb5-4a36-9d07-bf1a26672dec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr\" (UID: \"d4ea683d-ebb5-4a36-9d07-bf1a26672dec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"
Sep 15 19:15:47.126554 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.126520    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbfx8\" (UniqueName: \"kubernetes.io/projected/d4ea683d-ebb5-4a36-9d07-bf1a26672dec-kube-api-access-nbfx8\") pod \"payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr\" (UID: \"d4ea683d-ebb5-4a36-9d07-bf1a26672dec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"
Sep 15 19:15:47.126712 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.126582    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ea683d-ebb5-4a36-9d07-bf1a26672dec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr\" (UID: \"d4ea683d-ebb5-4a36-9d07-bf1a26672dec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"
Sep 15 19:15:47.126712 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.126649    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f86b20-ed93-45f2-bb0b-b94af151517c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5\" (UID: \"74f86b20-ed93-45f2-bb0b-b94af151517c\") " pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"
Sep 15 19:15:47.126712 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.126683    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn5tj\" (UniqueName: \"kubernetes.io/projected/74f86b20-ed93-45f2-bb0b-b94af151517c-kube-api-access-sn5tj\") pod \"payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5\" (UID: \"74f86b20-ed93-45f2-bb0b-b94af151517c\") " pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"
Sep 15 19:15:47.127397 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.127377    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ea683d-ebb5-4a36-9d07-bf1a26672dec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr\" (UID: \"d4ea683d-ebb5-4a36-9d07-bf1a26672dec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"
Sep 15 19:15:47.127442 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.127407    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f86b20-ed93-45f2-bb0b-b94af151517c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5\" (UID: \"74f86b20-ed93-45f2-bb0b-b94af151517c\") " pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"
Sep 15 19:15:47.134392 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.134363    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbfx8\" (UniqueName: \"kubernetes.io/projected/d4ea683d-ebb5-4a36-9d07-bf1a26672dec-kube-api-access-nbfx8\") pod \"payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr\" (UID: \"d4ea683d-ebb5-4a36-9d07-bf1a26672dec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"
Sep 15 19:15:47.134496 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.134419    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn5tj\" (UniqueName: \"kubernetes.io/projected/74f86b20-ed93-45f2-bb0b-b94af151517c-kube-api-access-sn5tj\") pod \"payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5\" (UID: \"74f86b20-ed93-45f2-bb0b-b94af151517c\") " pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"
Sep 15 19:15:47.250219 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.250150    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"
Sep 15 19:15:47.270485 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.270455    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"
Sep 15 19:15:47.523535 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.523510    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"]
Sep 15 19:15:47.524865 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:15:47.524837    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ea683d_ebb5_4a36_9d07_bf1a26672dec.slice/crio-85cd81092a0034c9cdd6df913307e0adb03dba483272f57c3f525241d5b6c377 WatchSource:0}: Error finding container 85cd81092a0034c9cdd6df913307e0adb03dba483272f57c3f525241d5b6c377: Status 404 returned error can't find the container with id 85cd81092a0034c9cdd6df913307e0adb03dba483272f57c3f525241d5b6c377
Sep 15 19:15:47.549503 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:47.549482    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"]
Sep 15 19:15:47.561346 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:15:47.561301    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f86b20_ed93_45f2_bb0b_b94af151517c.slice/crio-ace38159962b0e64eef96e7e4b681baa41ddfdeae92a7204922d622c26dcbfbf WatchSource:0}: Error finding container ace38159962b0e64eef96e7e4b681baa41ddfdeae92a7204922d622c26dcbfbf: Status 404 returned error can't find the container with id ace38159962b0e64eef96e7e4b681baa41ddfdeae92a7204922d622c26dcbfbf
Sep 15 19:15:48.398581 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:48.398539    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5" event={"ID":"74f86b20-ed93-45f2-bb0b-b94af151517c","Type":"ContainerStarted","Data":"38b6151b3a8aec98c32d206d4399793e3f412544222ba56836d6c3616900e914"}
Sep 15 19:15:48.399037 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:48.398598    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5" event={"ID":"74f86b20-ed93-45f2-bb0b-b94af151517c","Type":"ContainerStarted","Data":"ace38159962b0e64eef96e7e4b681baa41ddfdeae92a7204922d622c26dcbfbf"}
Sep 15 19:15:48.399037 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:48.398629    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"
Sep 15 19:15:48.400391 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:48.400358    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr" event={"ID":"d4ea683d-ebb5-4a36-9d07-bf1a26672dec","Type":"ContainerStarted","Data":"b8b30b291b7b6006e5ffedc36d938c548a838290275789b5e32787122d9ffd45"}
Sep 15 19:15:48.400516 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:48.400415    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr" event={"ID":"d4ea683d-ebb5-4a36-9d07-bf1a26672dec","Type":"ContainerStarted","Data":"85cd81092a0034c9cdd6df913307e0adb03dba483272f57c3f525241d5b6c377"}
Sep 15 19:15:48.414908 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:48.414861    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5" podStartSLOduration=2.414840895 podStartE2EDuration="2.414840895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:48.413332767 +0000 UTC m=+1510.690101624" watchObservedRunningTime="2026-09-15 19:15:48.414840895 +0000 UTC m=+1510.691609752"
Sep 15 19:15:48.429163 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:48.429125    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr" podStartSLOduration=2.429111847 podStartE2EDuration="2.429111847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:48.428030365 +0000 UTC m=+1510.704799222" watchObservedRunningTime="2026-09-15 19:15:48.429111847 +0000 UTC m=+1510.705880703"
Sep 15 19:15:49.405569 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:49.405532    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"
Sep 15 19:15:53.821305 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:53.821273    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:15:59.406908 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:15:59.406876    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"
Sep 15 19:16:00.412725 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:00.412650    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"
Sep 15 19:16:10.905963 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:10.905919    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:16:22.608740 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:22.608699    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:16:30.797789 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:30.797749    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"]
Sep 15 19:16:30.798591 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:30.798533    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8" podUID="360935f8-6868-4baf-8ca2-ac769dd652c1" containerName="payload-processing" containerID="cri-o://19f4adce975f677e91df4f2ae5232b241a2b24eaef6075a7f0d737b46524f415" gracePeriod=30
Sep 15 19:16:30.806660 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:30.806357    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"]
Sep 15 19:16:30.806660 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:30.806574    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7" podUID="f3103733-cfb4-4438-b0f9-0b6d4819de3b" containerName="payload-pre-processing" containerID="cri-o://12f2055cd2a80013c9e4bda1146f909e830dec49ddb9a9f575c90887a3a79086" gracePeriod=30
Sep 15 19:16:31.081733 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.081692    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"
Sep 15 19:16:31.106400 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.106361    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"
Sep 15 19:16:31.133712 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.133109    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/360935f8-6868-4baf-8ca2-ac769dd652c1-plugins-config-volume\") pod \"360935f8-6868-4baf-8ca2-ac769dd652c1\" (UID: \"360935f8-6868-4baf-8ca2-ac769dd652c1\") "
Sep 15 19:16:31.133712 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.133201    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqhs4\" (UniqueName: \"kubernetes.io/projected/360935f8-6868-4baf-8ca2-ac769dd652c1-kube-api-access-dqhs4\") pod \"360935f8-6868-4baf-8ca2-ac769dd652c1\" (UID: \"360935f8-6868-4baf-8ca2-ac769dd652c1\") "
Sep 15 19:16:31.133712 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.133276    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3103733-cfb4-4438-b0f9-0b6d4819de3b-plugins-config-volume\") pod \"f3103733-cfb4-4438-b0f9-0b6d4819de3b\" (UID: \"f3103733-cfb4-4438-b0f9-0b6d4819de3b\") "
Sep 15 19:16:31.133712 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.133365    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt7v7\" (UniqueName: \"kubernetes.io/projected/f3103733-cfb4-4438-b0f9-0b6d4819de3b-kube-api-access-mt7v7\") pod \"f3103733-cfb4-4438-b0f9-0b6d4819de3b\" (UID: \"f3103733-cfb4-4438-b0f9-0b6d4819de3b\") "
Sep 15 19:16:31.133712 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.133591    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/360935f8-6868-4baf-8ca2-ac769dd652c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "360935f8-6868-4baf-8ca2-ac769dd652c1" (UID: "360935f8-6868-4baf-8ca2-ac769dd652c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:16:31.133712 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.133664    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/360935f8-6868-4baf-8ca2-ac769dd652c1-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:31.134182 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.133772    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3103733-cfb4-4438-b0f9-0b6d4819de3b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3103733-cfb4-4438-b0f9-0b6d4819de3b" (UID: "f3103733-cfb4-4438-b0f9-0b6d4819de3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:16:31.135653 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.135621    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3103733-cfb4-4438-b0f9-0b6d4819de3b-kube-api-access-mt7v7" (OuterVolumeSpecName: "kube-api-access-mt7v7") pod "f3103733-cfb4-4438-b0f9-0b6d4819de3b" (UID: "f3103733-cfb4-4438-b0f9-0b6d4819de3b"). InnerVolumeSpecName "kube-api-access-mt7v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:16:31.135921 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.135899    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360935f8-6868-4baf-8ca2-ac769dd652c1-kube-api-access-dqhs4" (OuterVolumeSpecName: "kube-api-access-dqhs4") pod "360935f8-6868-4baf-8ca2-ac769dd652c1" (UID: "360935f8-6868-4baf-8ca2-ac769dd652c1"). InnerVolumeSpecName "kube-api-access-dqhs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:16:31.234514 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.234472    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt7v7\" (UniqueName: \"kubernetes.io/projected/f3103733-cfb4-4438-b0f9-0b6d4819de3b-kube-api-access-mt7v7\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:31.234514 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.234507    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqhs4\" (UniqueName: \"kubernetes.io/projected/360935f8-6868-4baf-8ca2-ac769dd652c1-kube-api-access-dqhs4\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:31.234514 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.234517    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3103733-cfb4-4438-b0f9-0b6d4819de3b-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:31.624287 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.624254    2550 generic.go:356] "Generic (PLEG): container finished" podID="360935f8-6868-4baf-8ca2-ac769dd652c1" containerID="19f4adce975f677e91df4f2ae5232b241a2b24eaef6075a7f0d737b46524f415" exitCode=0
Sep 15 19:16:31.624480 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.624295    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"
Sep 15 19:16:31.624480 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.624343    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8" event={"ID":"360935f8-6868-4baf-8ca2-ac769dd652c1","Type":"ContainerDied","Data":"19f4adce975f677e91df4f2ae5232b241a2b24eaef6075a7f0d737b46524f415"}
Sep 15 19:16:31.624480 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.624400    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8" event={"ID":"360935f8-6868-4baf-8ca2-ac769dd652c1","Type":"ContainerDied","Data":"28ac614b58cf3ba4dee7024f8ecc3a035e4d85bcaa948f965f7d3952f92bd300"}
Sep 15 19:16:31.624480 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.624426    2550 scope.go:117] "RemoveContainer" containerID="19f4adce975f677e91df4f2ae5232b241a2b24eaef6075a7f0d737b46524f415"
Sep 15 19:16:31.625936 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.625918    2550 generic.go:356] "Generic (PLEG): container finished" podID="f3103733-cfb4-4438-b0f9-0b6d4819de3b" containerID="12f2055cd2a80013c9e4bda1146f909e830dec49ddb9a9f575c90887a3a79086" exitCode=0
Sep 15 19:16:31.626043 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.625953    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"
Sep 15 19:16:31.626043 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.625959    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7" event={"ID":"f3103733-cfb4-4438-b0f9-0b6d4819de3b","Type":"ContainerDied","Data":"12f2055cd2a80013c9e4bda1146f909e830dec49ddb9a9f575c90887a3a79086"}
Sep 15 19:16:31.626131 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.626080    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7" event={"ID":"f3103733-cfb4-4438-b0f9-0b6d4819de3b","Type":"ContainerDied","Data":"fb8967db9418774a48738ddcdeb5adbb1f65b08920b664abb1eab31dc108c248"}
Sep 15 19:16:31.635142 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.635122    2550 scope.go:117] "RemoveContainer" containerID="19f4adce975f677e91df4f2ae5232b241a2b24eaef6075a7f0d737b46524f415"
Sep 15 19:16:31.635443 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:16:31.635425    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f4adce975f677e91df4f2ae5232b241a2b24eaef6075a7f0d737b46524f415\": container with ID starting with 19f4adce975f677e91df4f2ae5232b241a2b24eaef6075a7f0d737b46524f415 not found: ID does not exist" containerID="19f4adce975f677e91df4f2ae5232b241a2b24eaef6075a7f0d737b46524f415"
Sep 15 19:16:31.635519 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.635450    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f4adce975f677e91df4f2ae5232b241a2b24eaef6075a7f0d737b46524f415"} err="failed to get container status \"19f4adce975f677e91df4f2ae5232b241a2b24eaef6075a7f0d737b46524f415\": rpc error: code = NotFound desc = could not find container \"19f4adce975f677e91df4f2ae5232b241a2b24eaef6075a7f0d737b46524f415\": container with ID starting with 19f4adce975f677e91df4f2ae5232b241a2b24eaef6075a7f0d737b46524f415 not found: ID does not exist"
Sep 15 19:16:31.635519 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.635468    2550 scope.go:117] "RemoveContainer" containerID="12f2055cd2a80013c9e4bda1146f909e830dec49ddb9a9f575c90887a3a79086"
Sep 15 19:16:31.648213 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.647458    2550 scope.go:117] "RemoveContainer" containerID="12f2055cd2a80013c9e4bda1146f909e830dec49ddb9a9f575c90887a3a79086"
Sep 15 19:16:31.648923 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:16:31.648771    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f2055cd2a80013c9e4bda1146f909e830dec49ddb9a9f575c90887a3a79086\": container with ID starting with 12f2055cd2a80013c9e4bda1146f909e830dec49ddb9a9f575c90887a3a79086 not found: ID does not exist" containerID="12f2055cd2a80013c9e4bda1146f909e830dec49ddb9a9f575c90887a3a79086"
Sep 15 19:16:31.648923 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.648808    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f2055cd2a80013c9e4bda1146f909e830dec49ddb9a9f575c90887a3a79086"} err="failed to get container status \"12f2055cd2a80013c9e4bda1146f909e830dec49ddb9a9f575c90887a3a79086\": rpc error: code = NotFound desc = could not find container \"12f2055cd2a80013c9e4bda1146f909e830dec49ddb9a9f575c90887a3a79086\": container with ID starting with 12f2055cd2a80013c9e4bda1146f909e830dec49ddb9a9f575c90887a3a79086 not found: ID does not exist"
Sep 15 19:16:31.649821 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.649799    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"]
Sep 15 19:16:31.654530 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.654511    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-910e11-6cc584fc58-tjwh7"]
Sep 15 19:16:31.663875 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.663850    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"]
Sep 15 19:16:31.668291 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:31.668269    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-910e11-996975df7-287b8"]
Sep 15 19:16:32.302202 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:32.302166    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="360935f8-6868-4baf-8ca2-ac769dd652c1" path="/var/lib/kubelet/pods/360935f8-6868-4baf-8ca2-ac769dd652c1/volumes"
Sep 15 19:16:32.302705 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:32.302666    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3103733-cfb4-4438-b0f9-0b6d4819de3b" path="/var/lib/kubelet/pods/f3103733-cfb4-4438-b0f9-0b6d4819de3b/volumes"
Sep 15 19:16:33.712231 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:33.712193    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8927-2506b0"]
Sep 15 19:16:33.712792 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:33.712770    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="360935f8-6868-4baf-8ca2-ac769dd652c1" containerName="payload-processing"
Sep 15 19:16:33.712872 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:33.712794    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="360935f8-6868-4baf-8ca2-ac769dd652c1" containerName="payload-processing"
Sep 15 19:16:33.712872 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:33.712809    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3103733-cfb4-4438-b0f9-0b6d4819de3b" containerName="payload-pre-processing"
Sep 15 19:16:33.712872 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:33.712818    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3103733-cfb4-4438-b0f9-0b6d4819de3b" containerName="payload-pre-processing"
Sep 15 19:16:33.713028 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:33.712909    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3103733-cfb4-4438-b0f9-0b6d4819de3b" containerName="payload-pre-processing"
Sep 15 19:16:33.713028 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:33.712925    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="360935f8-6868-4baf-8ca2-ac769dd652c1" containerName="payload-processing"
Sep 15 19:16:33.717477 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:33.717456    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8927-2506b0"
Sep 15 19:16:33.719635 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:33.719608    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zrrnn\""
Sep 15 19:16:33.720996 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:33.720977    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8927-2506b0"]
Sep 15 19:16:33.755395 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:33.755362    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr6xt\" (UniqueName: \"kubernetes.io/projected/ea56b016-7973-474d-8b23-2f3fa407b346-kube-api-access-lr6xt\") pod \"test-curl-8927-2506b0\" (UID: \"ea56b016-7973-474d-8b23-2f3fa407b346\") " pod="opendatahub/test-curl-8927-2506b0"
Sep 15 19:16:33.856302 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:33.856243    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr6xt\" (UniqueName: \"kubernetes.io/projected/ea56b016-7973-474d-8b23-2f3fa407b346-kube-api-access-lr6xt\") pod \"test-curl-8927-2506b0\" (UID: \"ea56b016-7973-474d-8b23-2f3fa407b346\") " pod="opendatahub/test-curl-8927-2506b0"
Sep 15 19:16:33.863673 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:33.863650    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr6xt\" (UniqueName: \"kubernetes.io/projected/ea56b016-7973-474d-8b23-2f3fa407b346-kube-api-access-lr6xt\") pod \"test-curl-8927-2506b0\" (UID: \"ea56b016-7973-474d-8b23-2f3fa407b346\") " pod="opendatahub/test-curl-8927-2506b0"
Sep 15 19:16:34.027758 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:34.027730    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8927-2506b0"
Sep 15 19:16:34.297303 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:34.297278    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8927-2506b0"]
Sep 15 19:16:34.299303 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:16:34.299278    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea56b016_7973_474d_8b23_2f3fa407b346.slice/crio-f77af198a9c53b68cadf6b5964ee5ae6869d4b7cb1ab162f04e6b916d9876e1e WatchSource:0}: Error finding container f77af198a9c53b68cadf6b5964ee5ae6869d4b7cb1ab162f04e6b916d9876e1e: Status 404 returned error can't find the container with id f77af198a9c53b68cadf6b5964ee5ae6869d4b7cb1ab162f04e6b916d9876e1e
Sep 15 19:16:34.643434 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:34.643347    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8927-2506b0" event={"ID":"ea56b016-7973-474d-8b23-2f3fa407b346","Type":"ContainerStarted","Data":"217ed424745f3af4ae6e408ea669d70ed9915d91c6a3e14f87d4a99be97e0077"}
Sep 15 19:16:34.643434 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:34.643386    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8927-2506b0" event={"ID":"ea56b016-7973-474d-8b23-2f3fa407b346","Type":"ContainerStarted","Data":"f77af198a9c53b68cadf6b5964ee5ae6869d4b7cb1ab162f04e6b916d9876e1e"}
Sep 15 19:16:34.658421 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:34.658374    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8927-2506b0" podStartSLOduration=1.658359981 podStartE2EDuration="1.658359981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:16:34.657474179 +0000 UTC m=+1556.934243033" watchObservedRunningTime="2026-09-15 19:16:34.658359981 +0000 UTC m=+1556.935128836"
Sep 15 19:16:35.237361 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:35.237306    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8927-2506b0"]
Sep 15 19:16:35.240855 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:35.240831    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8927-2506b0"]
Sep 15 19:16:35.355587 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:35.355513    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8927-0895b7"]
Sep 15 19:16:35.355911 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:35.355898    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea56b016-7973-474d-8b23-2f3fa407b346" containerName="test-curl-8927-2506b0"
Sep 15 19:16:35.355953 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:35.355912    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea56b016-7973-474d-8b23-2f3fa407b346" containerName="test-curl-8927-2506b0"
Sep 15 19:16:35.355987 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:35.355974    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea56b016-7973-474d-8b23-2f3fa407b346" containerName="test-curl-8927-2506b0"
Sep 15 19:16:35.359104 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:35.359088    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8927-0895b7"
Sep 15 19:16:35.364387 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:35.364362    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8927-0895b7"]
Sep 15 19:16:35.470001 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:35.469974    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9htlr\" (UniqueName: \"kubernetes.io/projected/65166e4d-345b-4f18-ace6-fe5a5550a9d4-kube-api-access-9htlr\") pod \"test-curl-8927-0895b7\" (UID: \"65166e4d-345b-4f18-ace6-fe5a5550a9d4\") " pod="opendatahub/test-curl-8927-0895b7"
Sep 15 19:16:35.571236 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:35.571207    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9htlr\" (UniqueName: \"kubernetes.io/projected/65166e4d-345b-4f18-ace6-fe5a5550a9d4-kube-api-access-9htlr\") pod \"test-curl-8927-0895b7\" (UID: \"65166e4d-345b-4f18-ace6-fe5a5550a9d4\") " pod="opendatahub/test-curl-8927-0895b7"
Sep 15 19:16:35.578343 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:35.578300    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9htlr\" (UniqueName: \"kubernetes.io/projected/65166e4d-345b-4f18-ace6-fe5a5550a9d4-kube-api-access-9htlr\") pod \"test-curl-8927-0895b7\" (UID: \"65166e4d-345b-4f18-ace6-fe5a5550a9d4\") " pod="opendatahub/test-curl-8927-0895b7"
Sep 15 19:16:35.650887 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:16:35.650470    2550 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8927-2506b0" container="test-curl-8927-2506b0"
Sep 15 19:16:35.670173 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:35.670149    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8927-0895b7"
Sep 15 19:16:35.937559 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:35.937486    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8927-0895b7"]
Sep 15 19:16:35.940510 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:16:35.940484    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65166e4d_345b_4f18_ace6_fe5a5550a9d4.slice/crio-8090b2aef9e6cf4d59737ed559226773d99b4450d839ce9a01c687954f258351 WatchSource:0}: Error finding container 8090b2aef9e6cf4d59737ed559226773d99b4450d839ce9a01c687954f258351: Status 404 returned error can't find the container with id 8090b2aef9e6cf4d59737ed559226773d99b4450d839ce9a01c687954f258351
Sep 15 19:16:36.656415 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:36.656376    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8927-0895b7" event={"ID":"65166e4d-345b-4f18-ace6-fe5a5550a9d4","Type":"ContainerStarted","Data":"5d99213a16353d6c7a10d2adc2e7d14f6bff30bcbab594b8454112fb215d968e"}
Sep 15 19:16:36.656415 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:36.656420    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8927-0895b7" event={"ID":"65166e4d-345b-4f18-ace6-fe5a5550a9d4","Type":"ContainerStarted","Data":"8090b2aef9e6cf4d59737ed559226773d99b4450d839ce9a01c687954f258351"}
Sep 15 19:16:36.656899 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:36.656427    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8927-2506b0" podUID="ea56b016-7973-474d-8b23-2f3fa407b346" containerName="test-curl-8927-2506b0" containerID="cri-o://217ed424745f3af4ae6e408ea669d70ed9915d91c6a3e14f87d4a99be97e0077" gracePeriod=2
Sep 15 19:16:36.671323 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:36.671268    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8927-0895b7" podStartSLOduration=1.671254694 podStartE2EDuration="1.671254694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:16:36.669203227 +0000 UTC m=+1558.945972082" watchObservedRunningTime="2026-09-15 19:16:36.671254694 +0000 UTC m=+1558.948023549"
Sep 15 19:16:37.213303 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:37.213266    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"]
Sep 15 19:16:37.213537 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:37.213512    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t" podUID="88a42c8b-5966-4638-9799-60e7e99afe43" containerName="istio-proxy" containerID="cri-o://b22907354f916382ee50416fb30d19bebb38a0b83e2234cf9b54d4555220eb65" gracePeriod=30
Sep 15 19:16:37.319489 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:37.319412    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8927-0895b7"]
Sep 15 19:16:37.325813 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:37.325779    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8927-0895b7"]
Sep 15 19:16:37.661688 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:16:37.661615    2550 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8927-0895b7" container="test-curl-8927-0895b7"
Sep 15 19:16:37.792331 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:37.792258    2550 patch_prober.go:28] interesting pod/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 19:16:37.792511 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:37.792336    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t" podUID="88a42c8b-5966-4638-9799-60e7e99afe43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 19:16:38.667417 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:38.667386    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8927-0895b7" podUID="65166e4d-345b-4f18-ace6-fe5a5550a9d4" containerName="test-curl-8927-0895b7" containerID="cri-o://5d99213a16353d6c7a10d2adc2e7d14f6bff30bcbab594b8454112fb215d968e" gracePeriod=2
Sep 15 19:16:38.842028 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:38.842005    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8927-2506b0"
Sep 15 19:16:38.905301 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:38.905268    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr6xt\" (UniqueName: \"kubernetes.io/projected/ea56b016-7973-474d-8b23-2f3fa407b346-kube-api-access-lr6xt\") pod \"ea56b016-7973-474d-8b23-2f3fa407b346\" (UID: \"ea56b016-7973-474d-8b23-2f3fa407b346\") "
Sep 15 19:16:38.907284 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:38.907260    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea56b016-7973-474d-8b23-2f3fa407b346-kube-api-access-lr6xt" (OuterVolumeSpecName: "kube-api-access-lr6xt") pod "ea56b016-7973-474d-8b23-2f3fa407b346" (UID: "ea56b016-7973-474d-8b23-2f3fa407b346"). InnerVolumeSpecName "kube-api-access-lr6xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:16:39.012753 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:39.007753    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr6xt\" (UniqueName: \"kubernetes.io/projected/ea56b016-7973-474d-8b23-2f3fa407b346-kube-api-access-lr6xt\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:39.675265 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:39.675223    2550 generic.go:356] "Generic (PLEG): container finished" podID="ea56b016-7973-474d-8b23-2f3fa407b346" containerID="217ed424745f3af4ae6e408ea669d70ed9915d91c6a3e14f87d4a99be97e0077" exitCode=137
Sep 15 19:16:39.675728 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:39.675283    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8927-2506b0"
Sep 15 19:16:39.675728 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:39.675299    2550 scope.go:117] "RemoveContainer" containerID="217ed424745f3af4ae6e408ea669d70ed9915d91c6a3e14f87d4a99be97e0077"
Sep 15 19:16:39.684398 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:39.684381    2550 scope.go:117] "RemoveContainer" containerID="217ed424745f3af4ae6e408ea669d70ed9915d91c6a3e14f87d4a99be97e0077"
Sep 15 19:16:39.684670 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:16:39.684653    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217ed424745f3af4ae6e408ea669d70ed9915d91c6a3e14f87d4a99be97e0077\": container with ID starting with 217ed424745f3af4ae6e408ea669d70ed9915d91c6a3e14f87d4a99be97e0077 not found: ID does not exist" containerID="217ed424745f3af4ae6e408ea669d70ed9915d91c6a3e14f87d4a99be97e0077"
Sep 15 19:16:39.684738 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:39.684681    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217ed424745f3af4ae6e408ea669d70ed9915d91c6a3e14f87d4a99be97e0077"} err="failed to get container status \"217ed424745f3af4ae6e408ea669d70ed9915d91c6a3e14f87d4a99be97e0077\": rpc error: code = NotFound desc = could not find container \"217ed424745f3af4ae6e408ea669d70ed9915d91c6a3e14f87d4a99be97e0077\": container with ID starting with 217ed424745f3af4ae6e408ea669d70ed9915d91c6a3e14f87d4a99be97e0077 not found: ID does not exist"
Sep 15 19:16:39.685604 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:16:39.685581    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8927-2506b0\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="ea56b016-7973-474d-8b23-2f3fa407b346" pod="opendatahub/test-curl-8927-2506b0"
Sep 15 19:16:40.295271 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:40.295236    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea56b016-7973-474d-8b23-2f3fa407b346" path="/var/lib/kubelet/pods/ea56b016-7973-474d-8b23-2f3fa407b346/volumes"
Sep 15 19:16:40.875378 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:40.875359    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8927-0895b7"
Sep 15 19:16:40.926551 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:40.926485    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9htlr\" (UniqueName: \"kubernetes.io/projected/65166e4d-345b-4f18-ace6-fe5a5550a9d4-kube-api-access-9htlr\") pod \"65166e4d-345b-4f18-ace6-fe5a5550a9d4\" (UID: \"65166e4d-345b-4f18-ace6-fe5a5550a9d4\") "
Sep 15 19:16:40.928561 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:40.928538    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65166e4d-345b-4f18-ace6-fe5a5550a9d4-kube-api-access-9htlr" (OuterVolumeSpecName: "kube-api-access-9htlr") pod "65166e4d-345b-4f18-ace6-fe5a5550a9d4" (UID: "65166e4d-345b-4f18-ace6-fe5a5550a9d4"). InnerVolumeSpecName "kube-api-access-9htlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:16:41.027499 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:41.027442    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9htlr\" (UniqueName: \"kubernetes.io/projected/65166e4d-345b-4f18-ace6-fe5a5550a9d4-kube-api-access-9htlr\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:41.687362 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:41.687327    2550 generic.go:356] "Generic (PLEG): container finished" podID="65166e4d-345b-4f18-ace6-fe5a5550a9d4" containerID="5d99213a16353d6c7a10d2adc2e7d14f6bff30bcbab594b8454112fb215d968e" exitCode=137
Sep 15 19:16:41.687543 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:41.687377    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8927-0895b7"
Sep 15 19:16:41.687543 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:41.687417    2550 scope.go:117] "RemoveContainer" containerID="5d99213a16353d6c7a10d2adc2e7d14f6bff30bcbab594b8454112fb215d968e"
Sep 15 19:16:41.696417 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:41.696398    2550 scope.go:117] "RemoveContainer" containerID="5d99213a16353d6c7a10d2adc2e7d14f6bff30bcbab594b8454112fb215d968e"
Sep 15 19:16:41.696674 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:16:41.696655    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d99213a16353d6c7a10d2adc2e7d14f6bff30bcbab594b8454112fb215d968e\": container with ID starting with 5d99213a16353d6c7a10d2adc2e7d14f6bff30bcbab594b8454112fb215d968e not found: ID does not exist" containerID="5d99213a16353d6c7a10d2adc2e7d14f6bff30bcbab594b8454112fb215d968e"
Sep 15 19:16:41.696740 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:41.696685    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d99213a16353d6c7a10d2adc2e7d14f6bff30bcbab594b8454112fb215d968e"} err="failed to get container status \"5d99213a16353d6c7a10d2adc2e7d14f6bff30bcbab594b8454112fb215d968e\": rpc error: code = NotFound desc = could not find container \"5d99213a16353d6c7a10d2adc2e7d14f6bff30bcbab594b8454112fb215d968e\": container with ID starting with 5d99213a16353d6c7a10d2adc2e7d14f6bff30bcbab594b8454112fb215d968e not found: ID does not exist"
Sep 15 19:16:41.697539 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:16:41.697515    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8927-0895b7\" is forbidden: User \"system:node:ip-10-0-129-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-227.ec2.internal' and this object" podUID="65166e4d-345b-4f18-ace6-fe5a5550a9d4" pod="opendatahub/test-curl-8927-0895b7"
Sep 15 19:16:42.296726 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.296689    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65166e4d-345b-4f18-ace6-fe5a5550a9d4" path="/var/lib/kubelet/pods/65166e4d-345b-4f18-ace6-fe5a5550a9d4/volumes"
Sep 15 19:16:42.470614 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.470590    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:16:42.542952 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.542927    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a42c8b-5966-4638-9799-60e7e99afe43-istio-podinfo\") pod \"88a42c8b-5966-4638-9799-60e7e99afe43\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") "
Sep 15 19:16:42.543112 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.542963    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-istio-data\") pod \"88a42c8b-5966-4638-9799-60e7e99afe43\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") "
Sep 15 19:16:42.543112 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.543026    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-workload-socket\") pod \"88a42c8b-5966-4638-9799-60e7e99afe43\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") "
Sep 15 19:16:42.543112 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.543056    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-credential-socket\") pod \"88a42c8b-5966-4638-9799-60e7e99afe43\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") "
Sep 15 19:16:42.543112 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.543076    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-workload-certs\") pod \"88a42c8b-5966-4638-9799-60e7e99afe43\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") "
Sep 15 19:16:42.543112 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.543106    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44kfx\" (UniqueName: \"kubernetes.io/projected/88a42c8b-5966-4638-9799-60e7e99afe43-kube-api-access-44kfx\") pod \"88a42c8b-5966-4638-9799-60e7e99afe43\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") "
Sep 15 19:16:42.543405 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.543140    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a42c8b-5966-4638-9799-60e7e99afe43-istiod-ca-cert\") pod \"88a42c8b-5966-4638-9799-60e7e99afe43\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") "
Sep 15 19:16:42.543405 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.543182    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-istio-envoy\") pod \"88a42c8b-5966-4638-9799-60e7e99afe43\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") "
Sep 15 19:16:42.543405 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.543218    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a42c8b-5966-4638-9799-60e7e99afe43-istio-token\") pod \"88a42c8b-5966-4638-9799-60e7e99afe43\" (UID: \"88a42c8b-5966-4638-9799-60e7e99afe43\") "
Sep 15 19:16:42.543565 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.543393    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "88a42c8b-5966-4638-9799-60e7e99afe43" (UID: "88a42c8b-5966-4638-9799-60e7e99afe43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:16:42.543565 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.543459    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "88a42c8b-5966-4638-9799-60e7e99afe43" (UID: "88a42c8b-5966-4638-9799-60e7e99afe43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:16:42.543565 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.543432    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-istio-data" (OuterVolumeSpecName: "istio-data") pod "88a42c8b-5966-4638-9799-60e7e99afe43" (UID: "88a42c8b-5966-4638-9799-60e7e99afe43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:16:42.543718 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.543590    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-workload-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:42.543718 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.543607    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-istio-data\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:42.543718 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.543621    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-workload-socket\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:42.543718 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.543631    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a42c8b-5966-4638-9799-60e7e99afe43-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "88a42c8b-5966-4638-9799-60e7e99afe43" (UID: "88a42c8b-5966-4638-9799-60e7e99afe43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:16:42.543859 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.543719    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "88a42c8b-5966-4638-9799-60e7e99afe43" (UID: "88a42c8b-5966-4638-9799-60e7e99afe43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:16:42.545355 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.545294    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a42c8b-5966-4638-9799-60e7e99afe43-istio-token" (OuterVolumeSpecName: "istio-token") pod "88a42c8b-5966-4638-9799-60e7e99afe43" (UID: "88a42c8b-5966-4638-9799-60e7e99afe43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:16:42.545470 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.545374    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a42c8b-5966-4638-9799-60e7e99afe43-kube-api-access-44kfx" (OuterVolumeSpecName: "kube-api-access-44kfx") pod "88a42c8b-5966-4638-9799-60e7e99afe43" (UID: "88a42c8b-5966-4638-9799-60e7e99afe43"). InnerVolumeSpecName "kube-api-access-44kfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:16:42.545541 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.545524    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88a42c8b-5966-4638-9799-60e7e99afe43-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "88a42c8b-5966-4638-9799-60e7e99afe43" (UID: "88a42c8b-5966-4638-9799-60e7e99afe43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:16:42.545608 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.545589    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "88a42c8b-5966-4638-9799-60e7e99afe43" (UID: "88a42c8b-5966-4638-9799-60e7e99afe43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:16:42.644905 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.644871    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-credential-socket\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:42.644905 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.644898    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44kfx\" (UniqueName: \"kubernetes.io/projected/88a42c8b-5966-4638-9799-60e7e99afe43-kube-api-access-44kfx\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:42.644905 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.644908    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a42c8b-5966-4638-9799-60e7e99afe43-istiod-ca-cert\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:42.645109 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.644916    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a42c8b-5966-4638-9799-60e7e99afe43-istio-envoy\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:42.645109 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.644925    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a42c8b-5966-4638-9799-60e7e99afe43-istio-token\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:42.645109 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.644933    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a42c8b-5966-4638-9799-60e7e99afe43-istio-podinfo\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:42.695330 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.695287    2550 generic.go:356] "Generic (PLEG): container finished" podID="88a42c8b-5966-4638-9799-60e7e99afe43" containerID="b22907354f916382ee50416fb30d19bebb38a0b83e2234cf9b54d4555220eb65" exitCode=0
Sep 15 19:16:42.695457 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.695351    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t" event={"ID":"88a42c8b-5966-4638-9799-60e7e99afe43","Type":"ContainerDied","Data":"b22907354f916382ee50416fb30d19bebb38a0b83e2234cf9b54d4555220eb65"}
Sep 15 19:16:42.695457 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.695404    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t" event={"ID":"88a42c8b-5966-4638-9799-60e7e99afe43","Type":"ContainerDied","Data":"e4a5df7461cc69b789190a9491e3e9f2e85db7d2fd3258569c5254ef6c0bc005"}
Sep 15 19:16:42.695457 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.695426    2550 scope.go:117] "RemoveContainer" containerID="b22907354f916382ee50416fb30d19bebb38a0b83e2234cf9b54d4555220eb65"
Sep 15 19:16:42.695457 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.695365    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"
Sep 15 19:16:42.704350 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.704308    2550 scope.go:117] "RemoveContainer" containerID="e32ff4c54f44db03190b8363c40296c6077dfc59dc5b261bd78cb7f204449735"
Sep 15 19:16:42.712225 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.712207    2550 scope.go:117] "RemoveContainer" containerID="b22907354f916382ee50416fb30d19bebb38a0b83e2234cf9b54d4555220eb65"
Sep 15 19:16:42.712516 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:16:42.712496    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b22907354f916382ee50416fb30d19bebb38a0b83e2234cf9b54d4555220eb65\": container with ID starting with b22907354f916382ee50416fb30d19bebb38a0b83e2234cf9b54d4555220eb65 not found: ID does not exist" containerID="b22907354f916382ee50416fb30d19bebb38a0b83e2234cf9b54d4555220eb65"
Sep 15 19:16:42.712574 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.712525    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b22907354f916382ee50416fb30d19bebb38a0b83e2234cf9b54d4555220eb65"} err="failed to get container status \"b22907354f916382ee50416fb30d19bebb38a0b83e2234cf9b54d4555220eb65\": rpc error: code = NotFound desc = could not find container \"b22907354f916382ee50416fb30d19bebb38a0b83e2234cf9b54d4555220eb65\": container with ID starting with b22907354f916382ee50416fb30d19bebb38a0b83e2234cf9b54d4555220eb65 not found: ID does not exist"
Sep 15 19:16:42.712574 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.712544    2550 scope.go:117] "RemoveContainer" containerID="e32ff4c54f44db03190b8363c40296c6077dfc59dc5b261bd78cb7f204449735"
Sep 15 19:16:42.712778 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:16:42.712760    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e32ff4c54f44db03190b8363c40296c6077dfc59dc5b261bd78cb7f204449735\": container with ID starting with e32ff4c54f44db03190b8363c40296c6077dfc59dc5b261bd78cb7f204449735 not found: ID does not exist" containerID="e32ff4c54f44db03190b8363c40296c6077dfc59dc5b261bd78cb7f204449735"
Sep 15 19:16:42.712838 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.712785    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e32ff4c54f44db03190b8363c40296c6077dfc59dc5b261bd78cb7f204449735"} err="failed to get container status \"e32ff4c54f44db03190b8363c40296c6077dfc59dc5b261bd78cb7f204449735\": rpc error: code = NotFound desc = could not find container \"e32ff4c54f44db03190b8363c40296c6077dfc59dc5b261bd78cb7f204449735\": container with ID starting with e32ff4c54f44db03190b8363c40296c6077dfc59dc5b261bd78cb7f204449735 not found: ID does not exist"
Sep 15 19:16:42.716020 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.716000    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"]
Sep 15 19:16:42.719432 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:42.719411    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-910e11-openshift-default-6b47698bc4-4xb8t"]
Sep 15 19:16:43.007477 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:43.007399    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:16:44.295889 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:44.295853    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88a42c8b-5966-4638-9799-60e7e99afe43" path="/var/lib/kubelet/pods/88a42c8b-5966-4638-9799-60e7e99afe43/volumes"
Sep 15 19:16:53.203372 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.203336    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"]
Sep 15 19:16:53.203774 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.203588    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5" podUID="74f86b20-ed93-45f2-bb0b-b94af151517c" containerName="payload-processing" containerID="cri-o://38b6151b3a8aec98c32d206d4399793e3f412544222ba56836d6c3616900e914" gracePeriod=30
Sep 15 19:16:53.209460 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.209429    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"]
Sep 15 19:16:53.209681 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.209636    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr" podUID="d4ea683d-ebb5-4a36-9d07-bf1a26672dec" containerName="payload-pre-processing" containerID="cri-o://b8b30b291b7b6006e5ffedc36d938c548a838290275789b5e32787122d9ffd45" gracePeriod=30
Sep 15 19:16:53.495223 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.495199    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"
Sep 15 19:16:53.498027 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.498006    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"
Sep 15 19:16:53.645882 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.645848    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sn5tj\" (UniqueName: \"kubernetes.io/projected/74f86b20-ed93-45f2-bb0b-b94af151517c-kube-api-access-sn5tj\") pod \"74f86b20-ed93-45f2-bb0b-b94af151517c\" (UID: \"74f86b20-ed93-45f2-bb0b-b94af151517c\") "
Sep 15 19:16:53.646044 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.645989    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ea683d-ebb5-4a36-9d07-bf1a26672dec-plugins-config-volume\") pod \"d4ea683d-ebb5-4a36-9d07-bf1a26672dec\" (UID: \"d4ea683d-ebb5-4a36-9d07-bf1a26672dec\") "
Sep 15 19:16:53.646044 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.646031    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbfx8\" (UniqueName: \"kubernetes.io/projected/d4ea683d-ebb5-4a36-9d07-bf1a26672dec-kube-api-access-nbfx8\") pod \"d4ea683d-ebb5-4a36-9d07-bf1a26672dec\" (UID: \"d4ea683d-ebb5-4a36-9d07-bf1a26672dec\") "
Sep 15 19:16:53.646149 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.646051    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f86b20-ed93-45f2-bb0b-b94af151517c-plugins-config-volume\") pod \"74f86b20-ed93-45f2-bb0b-b94af151517c\" (UID: \"74f86b20-ed93-45f2-bb0b-b94af151517c\") "
Sep 15 19:16:53.646399 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.646365    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ea683d-ebb5-4a36-9d07-bf1a26672dec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4ea683d-ebb5-4a36-9d07-bf1a26672dec" (UID: "d4ea683d-ebb5-4a36-9d07-bf1a26672dec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:16:53.646518 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.646432    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f86b20-ed93-45f2-bb0b-b94af151517c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74f86b20-ed93-45f2-bb0b-b94af151517c" (UID: "74f86b20-ed93-45f2-bb0b-b94af151517c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:16:53.648044 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.648019    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ea683d-ebb5-4a36-9d07-bf1a26672dec-kube-api-access-nbfx8" (OuterVolumeSpecName: "kube-api-access-nbfx8") pod "d4ea683d-ebb5-4a36-9d07-bf1a26672dec" (UID: "d4ea683d-ebb5-4a36-9d07-bf1a26672dec"). InnerVolumeSpecName "kube-api-access-nbfx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:16:53.648153 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.648044    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f86b20-ed93-45f2-bb0b-b94af151517c-kube-api-access-sn5tj" (OuterVolumeSpecName: "kube-api-access-sn5tj") pod "74f86b20-ed93-45f2-bb0b-b94af151517c" (UID: "74f86b20-ed93-45f2-bb0b-b94af151517c"). InnerVolumeSpecName "kube-api-access-sn5tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:16:53.747626 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.747555    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbfx8\" (UniqueName: \"kubernetes.io/projected/d4ea683d-ebb5-4a36-9d07-bf1a26672dec-kube-api-access-nbfx8\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:53.747626 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.747582    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f86b20-ed93-45f2-bb0b-b94af151517c-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:53.747626 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.747591    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn5tj\" (UniqueName: \"kubernetes.io/projected/74f86b20-ed93-45f2-bb0b-b94af151517c-kube-api-access-sn5tj\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:53.747626 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.747600    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ea683d-ebb5-4a36-9d07-bf1a26672dec-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:16:53.757881 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.757850    2550 generic.go:356] "Generic (PLEG): container finished" podID="d4ea683d-ebb5-4a36-9d07-bf1a26672dec" containerID="b8b30b291b7b6006e5ffedc36d938c548a838290275789b5e32787122d9ffd45" exitCode=0
Sep 15 19:16:53.757999 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.757898    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"
Sep 15 19:16:53.757999 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.757935    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr" event={"ID":"d4ea683d-ebb5-4a36-9d07-bf1a26672dec","Type":"ContainerDied","Data":"b8b30b291b7b6006e5ffedc36d938c548a838290275789b5e32787122d9ffd45"}
Sep 15 19:16:53.757999 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.757985    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr" event={"ID":"d4ea683d-ebb5-4a36-9d07-bf1a26672dec","Type":"ContainerDied","Data":"85cd81092a0034c9cdd6df913307e0adb03dba483272f57c3f525241d5b6c377"}
Sep 15 19:16:53.758135 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.758006    2550 scope.go:117] "RemoveContainer" containerID="b8b30b291b7b6006e5ffedc36d938c548a838290275789b5e32787122d9ffd45"
Sep 15 19:16:53.759428 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.759405    2550 generic.go:356] "Generic (PLEG): container finished" podID="74f86b20-ed93-45f2-bb0b-b94af151517c" containerID="38b6151b3a8aec98c32d206d4399793e3f412544222ba56836d6c3616900e914" exitCode=0
Sep 15 19:16:53.759546 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.759450    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"
Sep 15 19:16:53.759546 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.759485    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5" event={"ID":"74f86b20-ed93-45f2-bb0b-b94af151517c","Type":"ContainerDied","Data":"38b6151b3a8aec98c32d206d4399793e3f412544222ba56836d6c3616900e914"}
Sep 15 19:16:53.759546 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.759527    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5" event={"ID":"74f86b20-ed93-45f2-bb0b-b94af151517c","Type":"ContainerDied","Data":"ace38159962b0e64eef96e7e4b681baa41ddfdeae92a7204922d622c26dcbfbf"}
Sep 15 19:16:53.768053 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.768019    2550 scope.go:117] "RemoveContainer" containerID="b8b30b291b7b6006e5ffedc36d938c548a838290275789b5e32787122d9ffd45"
Sep 15 19:16:53.768543 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:16:53.768523    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b30b291b7b6006e5ffedc36d938c548a838290275789b5e32787122d9ffd45\": container with ID starting with b8b30b291b7b6006e5ffedc36d938c548a838290275789b5e32787122d9ffd45 not found: ID does not exist" containerID="b8b30b291b7b6006e5ffedc36d938c548a838290275789b5e32787122d9ffd45"
Sep 15 19:16:53.768629 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.768550    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b30b291b7b6006e5ffedc36d938c548a838290275789b5e32787122d9ffd45"} err="failed to get container status \"b8b30b291b7b6006e5ffedc36d938c548a838290275789b5e32787122d9ffd45\": rpc error: code = NotFound desc = could not find container \"b8b30b291b7b6006e5ffedc36d938c548a838290275789b5e32787122d9ffd45\": container with ID starting with b8b30b291b7b6006e5ffedc36d938c548a838290275789b5e32787122d9ffd45 not found: ID does not exist"
Sep 15 19:16:53.768629 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.768566    2550 scope.go:117] "RemoveContainer" containerID="38b6151b3a8aec98c32d206d4399793e3f412544222ba56836d6c3616900e914"
Sep 15 19:16:53.776263 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.776244    2550 scope.go:117] "RemoveContainer" containerID="38b6151b3a8aec98c32d206d4399793e3f412544222ba56836d6c3616900e914"
Sep 15 19:16:53.776579 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:16:53.776562    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b6151b3a8aec98c32d206d4399793e3f412544222ba56836d6c3616900e914\": container with ID starting with 38b6151b3a8aec98c32d206d4399793e3f412544222ba56836d6c3616900e914 not found: ID does not exist" containerID="38b6151b3a8aec98c32d206d4399793e3f412544222ba56836d6c3616900e914"
Sep 15 19:16:53.776649 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.776588    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b6151b3a8aec98c32d206d4399793e3f412544222ba56836d6c3616900e914"} err="failed to get container status \"38b6151b3a8aec98c32d206d4399793e3f412544222ba56836d6c3616900e914\": rpc error: code = NotFound desc = could not find container \"38b6151b3a8aec98c32d206d4399793e3f412544222ba56836d6c3616900e914\": container with ID starting with 38b6151b3a8aec98c32d206d4399793e3f412544222ba56836d6c3616900e914 not found: ID does not exist"
Sep 15 19:16:53.780573 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.780551    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"]
Sep 15 19:16:53.784201 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.784182    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0a7ab7-799db6d58b-4jrpr"]
Sep 15 19:16:53.792523 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.792503    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"]
Sep 15 19:16:53.798290 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:53.798270    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0a7ab7-7bc9b64f77-gkmt5"]
Sep 15 19:16:54.302268 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:54.302230    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74f86b20-ed93-45f2-bb0b-b94af151517c" path="/var/lib/kubelet/pods/74f86b20-ed93-45f2-bb0b-b94af151517c/volumes"
Sep 15 19:16:54.302778 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:54.302757    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4ea683d-ebb5-4a36-9d07-bf1a26672dec" path="/var/lib/kubelet/pods/d4ea683d-ebb5-4a36-9d07-bf1a26672dec/volumes"
Sep 15 19:16:57.604473 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:16:57.604437    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:17:09.528400 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528362    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"]
Sep 15 19:17:09.528784 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528722    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f86b20-ed93-45f2-bb0b-b94af151517c" containerName="payload-processing"
Sep 15 19:17:09.528784 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528733    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f86b20-ed93-45f2-bb0b-b94af151517c" containerName="payload-processing"
Sep 15 19:17:09.528784 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528744    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a42c8b-5966-4638-9799-60e7e99afe43" containerName="istio-proxy"
Sep 15 19:17:09.528784 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528750    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a42c8b-5966-4638-9799-60e7e99afe43" containerName="istio-proxy"
Sep 15 19:17:09.528784 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528759    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65166e4d-345b-4f18-ace6-fe5a5550a9d4" containerName="test-curl-8927-0895b7"
Sep 15 19:17:09.528784 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528765    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="65166e4d-345b-4f18-ace6-fe5a5550a9d4" containerName="test-curl-8927-0895b7"
Sep 15 19:17:09.528784 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528774    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a42c8b-5966-4638-9799-60e7e99afe43" containerName="istio-proxy"
Sep 15 19:17:09.528784 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528780    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a42c8b-5966-4638-9799-60e7e99afe43" containerName="istio-proxy"
Sep 15 19:17:09.529073 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528794    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ea683d-ebb5-4a36-9d07-bf1a26672dec" containerName="payload-pre-processing"
Sep 15 19:17:09.529073 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528799    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ea683d-ebb5-4a36-9d07-bf1a26672dec" containerName="payload-pre-processing"
Sep 15 19:17:09.529073 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528854    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ea683d-ebb5-4a36-9d07-bf1a26672dec" containerName="payload-pre-processing"
Sep 15 19:17:09.529073 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528861    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="74f86b20-ed93-45f2-bb0b-b94af151517c" containerName="payload-processing"
Sep 15 19:17:09.529073 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528868    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a42c8b-5966-4638-9799-60e7e99afe43" containerName="istio-proxy"
Sep 15 19:17:09.529073 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528875    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a42c8b-5966-4638-9799-60e7e99afe43" containerName="istio-proxy"
Sep 15 19:17:09.529073 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.528881    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="65166e4d-345b-4f18-ace6-fe5a5550a9d4" containerName="test-curl-8927-0895b7"
Sep 15 19:17:09.531883 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.531863    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.535137 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.535109    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-7073b8\"/\"default-dockercfg-44jww\""
Sep 15 19:17:09.535274 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.535118    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-7073b8\"/\"openshift-service-ca.crt\""
Sep 15 19:17:09.535274 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.535211    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-7073b8\"/\"rate-test-model-7073b8-kserve-self-signed-certs\""
Sep 15 19:17:09.535414 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.535368    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-7073b8\"/\"kube-root-ca.crt\""
Sep 15 19:17:09.541877 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.541851    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"]
Sep 15 19:17:09.598330 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.598284    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-home\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.598534 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.598364    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrqr5\" (UniqueName: \"kubernetes.io/projected/26494a14-8d06-4dff-948d-2528ed2a5d8b-kube-api-access-qrqr5\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.598534 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.598398    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-model-cache\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.598534 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.598484    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26494a14-8d06-4dff-948d-2528ed2a5d8b-tls-certs\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.598534 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.598516    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-tmp-dir\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.598733 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.598544    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-dshm\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.699621 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.699586    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26494a14-8d06-4dff-948d-2528ed2a5d8b-tls-certs\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.699621 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.699626    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-tmp-dir\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.699884 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.699645    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-dshm\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.699884 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.699797    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-home\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.699884 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.699855    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrqr5\" (UniqueName: \"kubernetes.io/projected/26494a14-8d06-4dff-948d-2528ed2a5d8b-kube-api-access-qrqr5\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.700048 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.699887    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-model-cache\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.700100 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.700045    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-tmp-dir\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.700145 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.700127    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-home\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.700275 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.700255    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-model-cache\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.701970 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.701947    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-dshm\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.702351 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.702302    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26494a14-8d06-4dff-948d-2528ed2a5d8b-tls-certs\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.707405 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.707352    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrqr5\" (UniqueName: \"kubernetes.io/projected/26494a14-8d06-4dff-948d-2528ed2a5d8b-kube-api-access-qrqr5\") pod \"rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:09.842276 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:09.842201    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:10.115217 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:10.115191    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"]
Sep 15 19:17:10.117615 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:17:10.117589    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26494a14_8d06_4dff_948d_2528ed2a5d8b.slice/crio-83b9d62bdcf67b71a721bf6376a2fc82f0ab9237d2b51a6d085ebafd2ca75bb3 WatchSource:0}: Error finding container 83b9d62bdcf67b71a721bf6376a2fc82f0ab9237d2b51a6d085ebafd2ca75bb3: Status 404 returned error can't find the container with id 83b9d62bdcf67b71a721bf6376a2fc82f0ab9237d2b51a6d085ebafd2ca75bb3
Sep 15 19:17:10.848048 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:10.848009    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f" event={"ID":"26494a14-8d06-4dff-948d-2528ed2a5d8b","Type":"ContainerStarted","Data":"c20d9e86e5cc5114b404898cd8a5d93c934204b8478a706f05c6f9197c274f47"}
Sep 15 19:17:10.848048 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:10.848052    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f" event={"ID":"26494a14-8d06-4dff-948d-2528ed2a5d8b","Type":"ContainerStarted","Data":"83b9d62bdcf67b71a721bf6376a2fc82f0ab9237d2b51a6d085ebafd2ca75bb3"}
Sep 15 19:17:13.518886 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:13.518831    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f" podStartSLOduration=4.518813584 podStartE2EDuration="4.518813584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:17:10.864799838 +0000 UTC m=+1593.141568722" watchObservedRunningTime="2026-09-15 19:17:13.518813584 +0000 UTC m=+1595.795582448"
Sep 15 19:17:13.519297 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:13.519215    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:17:19.843267 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:19.843230    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:19.843749 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:19.843288    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:19.856038 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:19.856005    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:19.905825 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:19.905795    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:17:31.507990 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:31.507897    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:17:44.507565 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:44.507533    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:17:55.709105 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:17:55.709061    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:18:06.705508 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:06.705467    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:18:11.801487 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:11.801454    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"]
Sep 15 19:18:20.716817 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:20.716778    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:18:26.822546 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:26.822489    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f" podUID="26494a14-8d06-4dff-948d-2528ed2a5d8b" containerName="main" containerID="cri-o://c20d9e86e5cc5114b404898cd8a5d93c934204b8478a706f05c6f9197c274f47" gracePeriod=45
Sep 15 19:18:27.075151 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.075097    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:18:27.119253 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.119227    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-model-cache\") pod \"26494a14-8d06-4dff-948d-2528ed2a5d8b\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") "
Sep 15 19:18:27.119405 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.119268    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrqr5\" (UniqueName: \"kubernetes.io/projected/26494a14-8d06-4dff-948d-2528ed2a5d8b-kube-api-access-qrqr5\") pod \"26494a14-8d06-4dff-948d-2528ed2a5d8b\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") "
Sep 15 19:18:27.119405 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.119303    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-dshm\") pod \"26494a14-8d06-4dff-948d-2528ed2a5d8b\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") "
Sep 15 19:18:27.119405 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.119342    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-tmp-dir\") pod \"26494a14-8d06-4dff-948d-2528ed2a5d8b\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") "
Sep 15 19:18:27.119405 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.119363    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26494a14-8d06-4dff-948d-2528ed2a5d8b-tls-certs\") pod \"26494a14-8d06-4dff-948d-2528ed2a5d8b\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") "
Sep 15 19:18:27.119405 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.119404    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-home\") pod \"26494a14-8d06-4dff-948d-2528ed2a5d8b\" (UID: \"26494a14-8d06-4dff-948d-2528ed2a5d8b\") "
Sep 15 19:18:27.119673 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.119545    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-model-cache" (OuterVolumeSpecName: "model-cache") pod "26494a14-8d06-4dff-948d-2528ed2a5d8b" (UID: "26494a14-8d06-4dff-948d-2528ed2a5d8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:27.119673 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.119604    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26494a14-8d06-4dff-948d-2528ed2a5d8b" (UID: "26494a14-8d06-4dff-948d-2528ed2a5d8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:27.119792 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.119766    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-tmp-dir\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:18:27.119792 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.119770    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-home" (OuterVolumeSpecName: "home") pod "26494a14-8d06-4dff-948d-2528ed2a5d8b" (UID: "26494a14-8d06-4dff-948d-2528ed2a5d8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:27.119792 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.119785    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-model-cache\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:18:27.121349 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.121294    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-dshm" (OuterVolumeSpecName: "dshm") pod "26494a14-8d06-4dff-948d-2528ed2a5d8b" (UID: "26494a14-8d06-4dff-948d-2528ed2a5d8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:27.121349 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.121298    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26494a14-8d06-4dff-948d-2528ed2a5d8b-kube-api-access-qrqr5" (OuterVolumeSpecName: "kube-api-access-qrqr5") pod "26494a14-8d06-4dff-948d-2528ed2a5d8b" (UID: "26494a14-8d06-4dff-948d-2528ed2a5d8b"). InnerVolumeSpecName "kube-api-access-qrqr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:18:27.121477 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.121416    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26494a14-8d06-4dff-948d-2528ed2a5d8b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26494a14-8d06-4dff-948d-2528ed2a5d8b" (UID: "26494a14-8d06-4dff-948d-2528ed2a5d8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:18:27.220595 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.220570    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-home\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:18:27.220595 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.220592    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrqr5\" (UniqueName: \"kubernetes.io/projected/26494a14-8d06-4dff-948d-2528ed2a5d8b-kube-api-access-qrqr5\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:18:27.220725 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.220602    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26494a14-8d06-4dff-948d-2528ed2a5d8b-dshm\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:18:27.220725 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.220611    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26494a14-8d06-4dff-948d-2528ed2a5d8b-tls-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:18:27.239627 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.239596    2550 generic.go:356] "Generic (PLEG): container finished" podID="26494a14-8d06-4dff-948d-2528ed2a5d8b" containerID="c20d9e86e5cc5114b404898cd8a5d93c934204b8478a706f05c6f9197c274f47" exitCode=0
Sep 15 19:18:27.239730 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.239656    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"
Sep 15 19:18:27.239730 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.239678    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f" event={"ID":"26494a14-8d06-4dff-948d-2528ed2a5d8b","Type":"ContainerDied","Data":"c20d9e86e5cc5114b404898cd8a5d93c934204b8478a706f05c6f9197c274f47"}
Sep 15 19:18:27.239730 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.239722    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f" event={"ID":"26494a14-8d06-4dff-948d-2528ed2a5d8b","Type":"ContainerDied","Data":"83b9d62bdcf67b71a721bf6376a2fc82f0ab9237d2b51a6d085ebafd2ca75bb3"}
Sep 15 19:18:27.239858 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.239739    2550 scope.go:117] "RemoveContainer" containerID="c20d9e86e5cc5114b404898cd8a5d93c934204b8478a706f05c6f9197c274f47"
Sep 15 19:18:27.249221 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.249204    2550 scope.go:117] "RemoveContainer" containerID="c20d9e86e5cc5114b404898cd8a5d93c934204b8478a706f05c6f9197c274f47"
Sep 15 19:18:27.249559 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:18:27.249541    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c20d9e86e5cc5114b404898cd8a5d93c934204b8478a706f05c6f9197c274f47\": container with ID starting with c20d9e86e5cc5114b404898cd8a5d93c934204b8478a706f05c6f9197c274f47 not found: ID does not exist" containerID="c20d9e86e5cc5114b404898cd8a5d93c934204b8478a706f05c6f9197c274f47"
Sep 15 19:18:27.249628 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.249566    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c20d9e86e5cc5114b404898cd8a5d93c934204b8478a706f05c6f9197c274f47"} err="failed to get container status \"c20d9e86e5cc5114b404898cd8a5d93c934204b8478a706f05c6f9197c274f47\": rpc error: code = NotFound desc = could not find container \"c20d9e86e5cc5114b404898cd8a5d93c934204b8478a706f05c6f9197c274f47\": container with ID starting with c20d9e86e5cc5114b404898cd8a5d93c934204b8478a706f05c6f9197c274f47 not found: ID does not exist"
Sep 15 19:18:27.264324 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.264289    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"]
Sep 15 19:18:27.267636 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:27.267616    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/rate-test-model-7073b8-kserve-575cc8bd4c-jfl5f"]
Sep 15 19:18:28.295494 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:28.295466    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26494a14-8d06-4dff-948d-2528ed2a5d8b" path="/var/lib/kubelet/pods/26494a14-8d06-4dff-948d-2528ed2a5d8b/volumes"
Sep 15 19:18:48.378875 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.378842    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"]
Sep 15 19:18:48.379233 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.379198    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26494a14-8d06-4dff-948d-2528ed2a5d8b" containerName="main"
Sep 15 19:18:48.379233 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.379208    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="26494a14-8d06-4dff-948d-2528ed2a5d8b" containerName="main"
Sep 15 19:18:48.379330 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.379268    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="26494a14-8d06-4dff-948d-2528ed2a5d8b" containerName="main"
Sep 15 19:18:48.382290 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.382264    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"
Sep 15 19:18:48.384725 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.384702    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-25b1b3-dockercfg-dmxlh\""
Sep 15 19:18:48.384725 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.384722    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-25b1b3\""
Sep 15 19:18:48.395970 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.395950    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"]
Sep 15 19:18:48.500608 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.500575    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6a3635-da28-4ad9-b9c8-d8fdfcbcecde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh\" (UID: \"be6a3635-da28-4ad9-b9c8-d8fdfcbcecde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"
Sep 15 19:18:48.500790 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.500664    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsz6t\" (UniqueName: \"kubernetes.io/projected/be6a3635-da28-4ad9-b9c8-d8fdfcbcecde-kube-api-access-lsz6t\") pod \"payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh\" (UID: \"be6a3635-da28-4ad9-b9c8-d8fdfcbcecde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"
Sep 15 19:18:48.601701 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.601670    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6a3635-da28-4ad9-b9c8-d8fdfcbcecde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh\" (UID: \"be6a3635-da28-4ad9-b9c8-d8fdfcbcecde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"
Sep 15 19:18:48.601878 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.601730    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsz6t\" (UniqueName: \"kubernetes.io/projected/be6a3635-da28-4ad9-b9c8-d8fdfcbcecde-kube-api-access-lsz6t\") pod \"payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh\" (UID: \"be6a3635-da28-4ad9-b9c8-d8fdfcbcecde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"
Sep 15 19:18:48.602370 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.602350    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6a3635-da28-4ad9-b9c8-d8fdfcbcecde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh\" (UID: \"be6a3635-da28-4ad9-b9c8-d8fdfcbcecde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"
Sep 15 19:18:48.610188 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.610164    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsz6t\" (UniqueName: \"kubernetes.io/projected/be6a3635-da28-4ad9-b9c8-d8fdfcbcecde-kube-api-access-lsz6t\") pod \"payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh\" (UID: \"be6a3635-da28-4ad9-b9c8-d8fdfcbcecde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"
Sep 15 19:18:48.693545 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.693489    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"
Sep 15 19:18:48.964573 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:48.964545    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"]
Sep 15 19:18:48.966231 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:18:48.966202    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6a3635_da28_4ad9_b9c8_d8fdfcbcecde.slice/crio-fe6fb42af8c6343b509061636a06abc32248e8b2519ff3af623528bbbdd7a386 WatchSource:0}: Error finding container fe6fb42af8c6343b509061636a06abc32248e8b2519ff3af623528bbbdd7a386: Status 404 returned error can't find the container with id fe6fb42af8c6343b509061636a06abc32248e8b2519ff3af623528bbbdd7a386
Sep 15 19:18:49.349366 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:49.349307    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh" event={"ID":"be6a3635-da28-4ad9-b9c8-d8fdfcbcecde","Type":"ContainerStarted","Data":"326692b625d5b96fa223db7290b181f721b8db5f7a99c488d4208c8c0aae2517"}
Sep 15 19:18:49.349579 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:49.349383    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh" event={"ID":"be6a3635-da28-4ad9-b9c8-d8fdfcbcecde","Type":"ContainerStarted","Data":"fe6fb42af8c6343b509061636a06abc32248e8b2519ff3af623528bbbdd7a386"}
Sep 15 19:18:49.349579 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:49.349437    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"
Sep 15 19:18:49.365736 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:49.365692    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh" podStartSLOduration=1.365680101 podStartE2EDuration="1.365680101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:18:49.363825202 +0000 UTC m=+1691.640594075" watchObservedRunningTime="2026-09-15 19:18:49.365680101 +0000 UTC m=+1691.642448948"
Sep 15 19:18:53.317106 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:18:53.317069    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:19:00.356512 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:00.356432    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"
Sep 15 19:19:22.916835 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:22.916798    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:19:40.712411 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:40.712377    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:19:48.115601 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.115567    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"]
Sep 15 19:19:48.119469 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.119446    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.121922 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.121903    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-93b6ef-openshift-default-dockercfg-b5zk5\""
Sep 15 19:19:48.130775 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.130748    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"]
Sep 15 19:19:48.226117 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.226074    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2j5j\" (UniqueName: \"kubernetes.io/projected/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-kube-api-access-w2j5j\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.226406 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.226129    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-credential-socket\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.226406 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.226249    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-workload-certs\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.226406 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.226329    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-podinfo\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.226406 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.226368    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-workload-socket\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.226590 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.226434    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-envoy\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.226590 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.226478    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.226590 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.226520    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-data\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.226590 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.226543    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-token\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.327200 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.327165    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-podinfo\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.327200 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.327204    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-workload-socket\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.327428 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.327230    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-envoy\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.327428 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.327261    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.327428 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.327297    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-data\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.327428 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.327345    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-token\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.327428 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.327415    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2j5j\" (UniqueName: \"kubernetes.io/projected/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-kube-api-access-w2j5j\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.327670 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.327465    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-credential-socket\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.327670 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.327636    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-workload-socket\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.327772 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.327681    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-workload-certs\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.327772 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.327737    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-data\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.328001 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.327975    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-credential-socket\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.328056 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.328013    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-workload-certs\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.328056 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.328022    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.329600 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.329580    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-envoy\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.329730 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.329713    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-podinfo\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.337451 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.337389    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-token\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.337904 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.337881    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2j5j\" (UniqueName: \"kubernetes.io/projected/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-kube-api-access-w2j5j\") pod \"e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") " pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.431540 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.431462    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:48.707515 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.707438    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"]
Sep 15 19:19:48.710787 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:19:48.710761    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9949d4e0_0c2a_4c80_8e76_bd770ce6444c.slice/crio-d55f4166ce322da7805b32dbf0fab6ac63b57e33d54eb6af985eb8c4b03247cd WatchSource:0}: Error finding container d55f4166ce322da7805b32dbf0fab6ac63b57e33d54eb6af985eb8c4b03247cd: Status 404 returned error can't find the container with id d55f4166ce322da7805b32dbf0fab6ac63b57e33d54eb6af985eb8c4b03247cd
Sep 15 19:19:48.713077 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.713029    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:19:48.713170 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.713101    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:19:48.713170 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:48.713129    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:19:49.629627 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:49.629596    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n" event={"ID":"9949d4e0-0c2a-4c80-8e76-bd770ce6444c","Type":"ContainerStarted","Data":"e2aec9b1e105ccc9900da47f9c8f829e7a5dbee8ed73e5d7739b416e6348bbcf"}
Sep 15 19:19:49.629990 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:49.629641    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n" event={"ID":"9949d4e0-0c2a-4c80-8e76-bd770ce6444c","Type":"ContainerStarted","Data":"d55f4166ce322da7805b32dbf0fab6ac63b57e33d54eb6af985eb8c4b03247cd"}
Sep 15 19:19:49.651002 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:49.650959    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n" podStartSLOduration=1.65094639 podStartE2EDuration="1.65094639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:19:49.648762735 +0000 UTC m=+1751.925531792" watchObservedRunningTime="2026-09-15 19:19:49.65094639 +0000 UTC m=+1751.927715244"
Sep 15 19:19:50.431962 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:50.431931    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:50.437408 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:50.437378    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:50.636003 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:50.635970    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:50.636927 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:50.636901    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:19:55.087442 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:55.087405    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"]
Sep 15 19:19:55.090867 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:55.090851    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"
Sep 15 19:19:55.093325 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:55.093284    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-93b6ef\""
Sep 15 19:19:55.093325 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:55.093304    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-93b6ef-dockercfg-bcfpr\""
Sep 15 19:19:55.102058 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:55.102036    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"]
Sep 15 19:19:55.190724 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:55.190689    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25ca180-31bf-4e3f-bdfa-08531f231196-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss\" (UID: \"a25ca180-31bf-4e3f-bdfa-08531f231196\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"
Sep 15 19:19:55.190899 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:55.190749    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p9cb\" (UniqueName: \"kubernetes.io/projected/a25ca180-31bf-4e3f-bdfa-08531f231196-kube-api-access-5p9cb\") pod \"payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss\" (UID: \"a25ca180-31bf-4e3f-bdfa-08531f231196\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"
Sep 15 19:19:55.291870 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:55.291836    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25ca180-31bf-4e3f-bdfa-08531f231196-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss\" (UID: \"a25ca180-31bf-4e3f-bdfa-08531f231196\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"
Sep 15 19:19:55.292038 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:55.291895    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p9cb\" (UniqueName: \"kubernetes.io/projected/a25ca180-31bf-4e3f-bdfa-08531f231196-kube-api-access-5p9cb\") pod \"payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss\" (UID: \"a25ca180-31bf-4e3f-bdfa-08531f231196\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"
Sep 15 19:19:55.292593 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:55.292566    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25ca180-31bf-4e3f-bdfa-08531f231196-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss\" (UID: \"a25ca180-31bf-4e3f-bdfa-08531f231196\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"
Sep 15 19:19:55.299277 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:55.299248    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p9cb\" (UniqueName: \"kubernetes.io/projected/a25ca180-31bf-4e3f-bdfa-08531f231196-kube-api-access-5p9cb\") pod \"payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss\" (UID: \"a25ca180-31bf-4e3f-bdfa-08531f231196\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"
Sep 15 19:19:55.402720 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:55.402639    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"
Sep 15 19:19:55.673400 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:55.673362    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"]
Sep 15 19:19:55.675724 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:19:55.675692    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25ca180_31bf_4e3f_bdfa_08531f231196.slice/crio-23c9a97ba07dc38120c535f7c2be5407cae42a2ecd9399903b9cb541959a25c0 WatchSource:0}: Error finding container 23c9a97ba07dc38120c535f7c2be5407cae42a2ecd9399903b9cb541959a25c0: Status 404 returned error can't find the container with id 23c9a97ba07dc38120c535f7c2be5407cae42a2ecd9399903b9cb541959a25c0
Sep 15 19:19:56.668401 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:56.668360    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss" event={"ID":"a25ca180-31bf-4e3f-bdfa-08531f231196","Type":"ContainerStarted","Data":"8356044b7228e82d46e138ce1ba75b27a6cd5afa9ec38ad219393076943887e6"}
Sep 15 19:19:56.668793 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:56.668416    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss" event={"ID":"a25ca180-31bf-4e3f-bdfa-08531f231196","Type":"ContainerStarted","Data":"23c9a97ba07dc38120c535f7c2be5407cae42a2ecd9399903b9cb541959a25c0"}
Sep 15 19:19:56.668793 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:56.668439    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"
Sep 15 19:19:56.684344 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:56.684283    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss" podStartSLOduration=1.684270568 podStartE2EDuration="1.684270568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:19:56.682997074 +0000 UTC m=+1758.959765932" watchObservedRunningTime="2026-09-15 19:19:56.684270568 +0000 UTC m=+1758.961039477"
Sep 15 19:19:59.518796 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:19:59.518764    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:20:07.676236 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:07.676156    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"
Sep 15 19:20:10.752304 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:10.752271    2550 generic.go:356] "Generic (PLEG): container finished" podID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" containerID="e2aec9b1e105ccc9900da47f9c8f829e7a5dbee8ed73e5d7739b416e6348bbcf" exitCode=0
Sep 15 19:20:10.752770 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:10.752342    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n" event={"ID":"9949d4e0-0c2a-4c80-8e76-bd770ce6444c","Type":"ContainerDied","Data":"e2aec9b1e105ccc9900da47f9c8f829e7a5dbee8ed73e5d7739b416e6348bbcf"}
Sep 15 19:20:10.752770 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:10.752727    2550 scope.go:117] "RemoveContainer" containerID="e2aec9b1e105ccc9900da47f9c8f829e7a5dbee8ed73e5d7739b416e6348bbcf"
Sep 15 19:20:10.753817 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:10.753781    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:20:10.753927 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:10.753861    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:20:10.753927 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:10.753892    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 15 19:20:11.759179 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:11.759141    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n" event={"ID":"9949d4e0-0c2a-4c80-8e76-bd770ce6444c","Type":"ContainerStarted","Data":"b7b53eec562faaf93b8a6d3cbfdec7f35d652335b90cea938bf48a83a8c7ba6c"}
Sep 15 19:20:12.432477 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:12.432437    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:20:12.432739 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:12.432716    2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 15 19:20:12.432827 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:12.432771    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n" podUID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 15 19:20:13.432221 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:13.432186    2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 15 19:20:13.432627 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:13.432242    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n" podUID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 15 19:20:14.432805 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:14.432763    2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 15 19:20:14.433297 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:14.432843    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n" podUID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 15 19:20:15.432336 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:15.432278    2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 15 19:20:15.432544 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:15.432367    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n" podUID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 15 19:20:16.431897 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:16.431861    2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 15 19:20:16.432286 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:16.431914    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n" podUID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 15 19:20:17.436767 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:17.436737    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:20:17.437228 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:17.437101    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:20:17.437972 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:17.437949    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:20:23.613118 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:23.613080    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"]
Sep 15 19:20:23.613657 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:23.613299    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss" podUID="a25ca180-31bf-4e3f-bdfa-08531f231196" containerName="payload-processing" containerID="cri-o://8356044b7228e82d46e138ce1ba75b27a6cd5afa9ec38ad219393076943887e6" gracePeriod=30
Sep 15 19:20:23.821509 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:23.821462    2550 generic.go:356] "Generic (PLEG): container finished" podID="a25ca180-31bf-4e3f-bdfa-08531f231196" containerID="8356044b7228e82d46e138ce1ba75b27a6cd5afa9ec38ad219393076943887e6" exitCode=0
Sep 15 19:20:23.821676 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:23.821533    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss" event={"ID":"a25ca180-31bf-4e3f-bdfa-08531f231196","Type":"ContainerDied","Data":"8356044b7228e82d46e138ce1ba75b27a6cd5afa9ec38ad219393076943887e6"}
Sep 15 19:20:23.875621 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:23.875567    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"
Sep 15 19:20:23.955155 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:23.955120    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25ca180-31bf-4e3f-bdfa-08531f231196-plugins-config-volume\") pod \"a25ca180-31bf-4e3f-bdfa-08531f231196\" (UID: \"a25ca180-31bf-4e3f-bdfa-08531f231196\") "
Sep 15 19:20:23.955349 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:23.955213    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p9cb\" (UniqueName: \"kubernetes.io/projected/a25ca180-31bf-4e3f-bdfa-08531f231196-kube-api-access-5p9cb\") pod \"a25ca180-31bf-4e3f-bdfa-08531f231196\" (UID: \"a25ca180-31bf-4e3f-bdfa-08531f231196\") "
Sep 15 19:20:23.955566 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:23.955537    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25ca180-31bf-4e3f-bdfa-08531f231196-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a25ca180-31bf-4e3f-bdfa-08531f231196" (UID: "a25ca180-31bf-4e3f-bdfa-08531f231196"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:20:23.957392 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:23.957361    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25ca180-31bf-4e3f-bdfa-08531f231196-kube-api-access-5p9cb" (OuterVolumeSpecName: "kube-api-access-5p9cb") pod "a25ca180-31bf-4e3f-bdfa-08531f231196" (UID: "a25ca180-31bf-4e3f-bdfa-08531f231196"). InnerVolumeSpecName "kube-api-access-5p9cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:24.056149 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:24.056117    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p9cb\" (UniqueName: \"kubernetes.io/projected/a25ca180-31bf-4e3f-bdfa-08531f231196-kube-api-access-5p9cb\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:20:24.056149 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:24.056144    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25ca180-31bf-4e3f-bdfa-08531f231196-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:20:24.827678 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:24.827650    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"
Sep 15 19:20:24.828075 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:24.827682    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss" event={"ID":"a25ca180-31bf-4e3f-bdfa-08531f231196","Type":"ContainerDied","Data":"23c9a97ba07dc38120c535f7c2be5407cae42a2ecd9399903b9cb541959a25c0"}
Sep 15 19:20:24.828075 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:24.827735    2550 scope.go:117] "RemoveContainer" containerID="8356044b7228e82d46e138ce1ba75b27a6cd5afa9ec38ad219393076943887e6"
Sep 15 19:20:24.843225 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:24.843201    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"]
Sep 15 19:20:24.850764 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:24.850740    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-93b6ef-798fcc449-ftnss"]
Sep 15 19:20:26.295005 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:26.294976    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a25ca180-31bf-4e3f-bdfa-08531f231196" path="/var/lib/kubelet/pods/a25ca180-31bf-4e3f-bdfa-08531f231196/volumes"
Sep 15 19:20:28.118073 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:28.118038    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:20:30.107838 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:30.107808    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"]
Sep 15 19:20:30.108305 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:30.108095    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n" podUID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" containerName="istio-proxy" containerID="cri-o://b7b53eec562faaf93b8a6d3cbfdec7f35d652335b90cea938bf48a83a8c7ba6c" gracePeriod=30
Sep 15 19:20:32.440226 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:32.440192    2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 19:20:32.440615 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:32.440249    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n" podUID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 19:20:35.376213 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.376156    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:20:35.456453 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.456413    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-envoy\") pod \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") "
Sep 15 19:20:35.456453 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.456460    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-workload-certs\") pod \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") "
Sep 15 19:20:35.456658 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.456508    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istiod-ca-cert\") pod \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") "
Sep 15 19:20:35.456658 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.456532    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-token\") pod \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") "
Sep 15 19:20:35.456658 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.456558    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2j5j\" (UniqueName: \"kubernetes.io/projected/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-kube-api-access-w2j5j\") pod \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") "
Sep 15 19:20:35.456658 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.456597    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-workload-socket\") pod \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") "
Sep 15 19:20:35.456658 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.456641    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-credential-socket\") pod \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") "
Sep 15 19:20:35.456925 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.456663    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-data\") pod \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") "
Sep 15 19:20:35.456925 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.456736    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-podinfo\") pod \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\" (UID: \"9949d4e0-0c2a-4c80-8e76-bd770ce6444c\") "
Sep 15 19:20:35.456925 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.456810    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9949d4e0-0c2a-4c80-8e76-bd770ce6444c" (UID: "9949d4e0-0c2a-4c80-8e76-bd770ce6444c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:35.457093 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.456925    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9949d4e0-0c2a-4c80-8e76-bd770ce6444c" (UID: "9949d4e0-0c2a-4c80-8e76-bd770ce6444c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:35.457093 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.456950    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9949d4e0-0c2a-4c80-8e76-bd770ce6444c" (UID: "9949d4e0-0c2a-4c80-8e76-bd770ce6444c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:20:35.457093 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.457032    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9949d4e0-0c2a-4c80-8e76-bd770ce6444c" (UID: "9949d4e0-0c2a-4c80-8e76-bd770ce6444c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:35.457247 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.457146    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-workload-socket\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:20:35.457247 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.457156    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-data" (OuterVolumeSpecName: "istio-data") pod "9949d4e0-0c2a-4c80-8e76-bd770ce6444c" (UID: "9949d4e0-0c2a-4c80-8e76-bd770ce6444c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:35.457247 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.457163    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-credential-socket\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:20:35.457247 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.457194    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-workload-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:20:35.457247 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.457210    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istiod-ca-cert\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:20:35.458880 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.458853    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9949d4e0-0c2a-4c80-8e76-bd770ce6444c" (UID: "9949d4e0-0c2a-4c80-8e76-bd770ce6444c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:20:35.458880 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.458859    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-token" (OuterVolumeSpecName: "istio-token") pod "9949d4e0-0c2a-4c80-8e76-bd770ce6444c" (UID: "9949d4e0-0c2a-4c80-8e76-bd770ce6444c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:35.459068 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.459047    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9949d4e0-0c2a-4c80-8e76-bd770ce6444c" (UID: "9949d4e0-0c2a-4c80-8e76-bd770ce6444c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:35.459116 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.459047    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-kube-api-access-w2j5j" (OuterVolumeSpecName: "kube-api-access-w2j5j") pod "9949d4e0-0c2a-4c80-8e76-bd770ce6444c" (UID: "9949d4e0-0c2a-4c80-8e76-bd770ce6444c"). InnerVolumeSpecName "kube-api-access-w2j5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:35.558346 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.558299    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-token\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:20:35.558346 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.558347    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2j5j\" (UniqueName: \"kubernetes.io/projected/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-kube-api-access-w2j5j\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:20:35.558521 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.558358    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-data\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:20:35.558521 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.558367    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-podinfo\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:20:35.558521 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.558375    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9949d4e0-0c2a-4c80-8e76-bd770ce6444c-istio-envoy\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:20:35.891971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.891937    2550 generic.go:356] "Generic (PLEG): container finished" podID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" containerID="b7b53eec562faaf93b8a6d3cbfdec7f35d652335b90cea938bf48a83a8c7ba6c" exitCode=0
Sep 15 19:20:35.892149 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.891989    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"
Sep 15 19:20:35.892149 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.892031    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n" event={"ID":"9949d4e0-0c2a-4c80-8e76-bd770ce6444c","Type":"ContainerDied","Data":"b7b53eec562faaf93b8a6d3cbfdec7f35d652335b90cea938bf48a83a8c7ba6c"}
Sep 15 19:20:35.892149 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.892089    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n" event={"ID":"9949d4e0-0c2a-4c80-8e76-bd770ce6444c","Type":"ContainerDied","Data":"d55f4166ce322da7805b32dbf0fab6ac63b57e33d54eb6af985eb8c4b03247cd"}
Sep 15 19:20:35.892149 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.892114    2550 scope.go:117] "RemoveContainer" containerID="b7b53eec562faaf93b8a6d3cbfdec7f35d652335b90cea938bf48a83a8c7ba6c"
Sep 15 19:20:35.901890 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.901870    2550 scope.go:117] "RemoveContainer" containerID="e2aec9b1e105ccc9900da47f9c8f829e7a5dbee8ed73e5d7739b416e6348bbcf"
Sep 15 19:20:35.909277 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.909262    2550 scope.go:117] "RemoveContainer" containerID="b7b53eec562faaf93b8a6d3cbfdec7f35d652335b90cea938bf48a83a8c7ba6c"
Sep 15 19:20:35.909526 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:20:35.909509    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b53eec562faaf93b8a6d3cbfdec7f35d652335b90cea938bf48a83a8c7ba6c\": container with ID starting with b7b53eec562faaf93b8a6d3cbfdec7f35d652335b90cea938bf48a83a8c7ba6c not found: ID does not exist" containerID="b7b53eec562faaf93b8a6d3cbfdec7f35d652335b90cea938bf48a83a8c7ba6c"
Sep 15 19:20:35.909588 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.909532    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b53eec562faaf93b8a6d3cbfdec7f35d652335b90cea938bf48a83a8c7ba6c"} err="failed to get container status \"b7b53eec562faaf93b8a6d3cbfdec7f35d652335b90cea938bf48a83a8c7ba6c\": rpc error: code = NotFound desc = could not find container \"b7b53eec562faaf93b8a6d3cbfdec7f35d652335b90cea938bf48a83a8c7ba6c\": container with ID starting with b7b53eec562faaf93b8a6d3cbfdec7f35d652335b90cea938bf48a83a8c7ba6c not found: ID does not exist"
Sep 15 19:20:35.909588 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.909547    2550 scope.go:117] "RemoveContainer" containerID="e2aec9b1e105ccc9900da47f9c8f829e7a5dbee8ed73e5d7739b416e6348bbcf"
Sep 15 19:20:35.909744 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:20:35.909729    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2aec9b1e105ccc9900da47f9c8f829e7a5dbee8ed73e5d7739b416e6348bbcf\": container with ID starting with e2aec9b1e105ccc9900da47f9c8f829e7a5dbee8ed73e5d7739b416e6348bbcf not found: ID does not exist" containerID="e2aec9b1e105ccc9900da47f9c8f829e7a5dbee8ed73e5d7739b416e6348bbcf"
Sep 15 19:20:35.909787 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.909748    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2aec9b1e105ccc9900da47f9c8f829e7a5dbee8ed73e5d7739b416e6348bbcf"} err="failed to get container status \"e2aec9b1e105ccc9900da47f9c8f829e7a5dbee8ed73e5d7739b416e6348bbcf\": rpc error: code = NotFound desc = could not find container \"e2aec9b1e105ccc9900da47f9c8f829e7a5dbee8ed73e5d7739b416e6348bbcf\": container with ID starting with e2aec9b1e105ccc9900da47f9c8f829e7a5dbee8ed73e5d7739b416e6348bbcf not found: ID does not exist"
Sep 15 19:20:35.925456 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.925431    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"]
Sep 15 19:20:35.929472 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:35.929450    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-93b6ef-openshift-default-7d4d79f9cd-n8b2n"]
Sep 15 19:20:36.295859 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:36.295827    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" path="/var/lib/kubelet/pods/9949d4e0-0c2a-4c80-8e76-bd770ce6444c/volumes"
Sep 15 19:20:47.107347 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:47.107307    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:20:59.904872 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:20:59.904838    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:21:49.185142 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:49.185104    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"]
Sep 15 19:21:49.185779 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:49.185383    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh" podUID="be6a3635-da28-4ad9-b9c8-d8fdfcbcecde" containerName="payload-pre-processing" containerID="cri-o://326692b625d5b96fa223db7290b181f721b8db5f7a99c488d4208c8c0aae2517" gracePeriod=30
Sep 15 19:21:49.448365 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:49.448299    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"
Sep 15 19:21:49.507578 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:49.507548    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsz6t\" (UniqueName: \"kubernetes.io/projected/be6a3635-da28-4ad9-b9c8-d8fdfcbcecde-kube-api-access-lsz6t\") pod \"be6a3635-da28-4ad9-b9c8-d8fdfcbcecde\" (UID: \"be6a3635-da28-4ad9-b9c8-d8fdfcbcecde\") "
Sep 15 19:21:49.507739 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:49.507650    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6a3635-da28-4ad9-b9c8-d8fdfcbcecde-plugins-config-volume\") pod \"be6a3635-da28-4ad9-b9c8-d8fdfcbcecde\" (UID: \"be6a3635-da28-4ad9-b9c8-d8fdfcbcecde\") "
Sep 15 19:21:49.507992 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:49.507971    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6a3635-da28-4ad9-b9c8-d8fdfcbcecde-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be6a3635-da28-4ad9-b9c8-d8fdfcbcecde" (UID: "be6a3635-da28-4ad9-b9c8-d8fdfcbcecde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:49.509712 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:49.509679    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6a3635-da28-4ad9-b9c8-d8fdfcbcecde-kube-api-access-lsz6t" (OuterVolumeSpecName: "kube-api-access-lsz6t") pod "be6a3635-da28-4ad9-b9c8-d8fdfcbcecde" (UID: "be6a3635-da28-4ad9-b9c8-d8fdfcbcecde"). InnerVolumeSpecName "kube-api-access-lsz6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:49.608299 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:49.608251    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsz6t\" (UniqueName: \"kubernetes.io/projected/be6a3635-da28-4ad9-b9c8-d8fdfcbcecde-kube-api-access-lsz6t\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:21:49.608299 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:49.608291    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6a3635-da28-4ad9-b9c8-d8fdfcbcecde-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:21:50.249917 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:50.249879    2550 generic.go:356] "Generic (PLEG): container finished" podID="be6a3635-da28-4ad9-b9c8-d8fdfcbcecde" containerID="326692b625d5b96fa223db7290b181f721b8db5f7a99c488d4208c8c0aae2517" exitCode=0
Sep 15 19:21:50.250342 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:50.249934    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"
Sep 15 19:21:50.250342 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:50.249944    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh" event={"ID":"be6a3635-da28-4ad9-b9c8-d8fdfcbcecde","Type":"ContainerDied","Data":"326692b625d5b96fa223db7290b181f721b8db5f7a99c488d4208c8c0aae2517"}
Sep 15 19:21:50.250342 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:50.250015    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh" event={"ID":"be6a3635-da28-4ad9-b9c8-d8fdfcbcecde","Type":"ContainerDied","Data":"fe6fb42af8c6343b509061636a06abc32248e8b2519ff3af623528bbbdd7a386"}
Sep 15 19:21:50.250342 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:50.250042    2550 scope.go:117] "RemoveContainer" containerID="326692b625d5b96fa223db7290b181f721b8db5f7a99c488d4208c8c0aae2517"
Sep 15 19:21:50.260304 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:50.260286    2550 scope.go:117] "RemoveContainer" containerID="326692b625d5b96fa223db7290b181f721b8db5f7a99c488d4208c8c0aae2517"
Sep 15 19:21:50.260580 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:21:50.260557    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"326692b625d5b96fa223db7290b181f721b8db5f7a99c488d4208c8c0aae2517\": container with ID starting with 326692b625d5b96fa223db7290b181f721b8db5f7a99c488d4208c8c0aae2517 not found: ID does not exist" containerID="326692b625d5b96fa223db7290b181f721b8db5f7a99c488d4208c8c0aae2517"
Sep 15 19:21:50.260644 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:50.260592    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"326692b625d5b96fa223db7290b181f721b8db5f7a99c488d4208c8c0aae2517"} err="failed to get container status \"326692b625d5b96fa223db7290b181f721b8db5f7a99c488d4208c8c0aae2517\": rpc error: code = NotFound desc = could not find container \"326692b625d5b96fa223db7290b181f721b8db5f7a99c488d4208c8c0aae2517\": container with ID starting with 326692b625d5b96fa223db7290b181f721b8db5f7a99c488d4208c8c0aae2517 not found: ID does not exist"
Sep 15 19:21:50.270916 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:50.270892    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"]
Sep 15 19:21:50.274873 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:50.274851    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-25b1b3-5967ccd554-tjbbh"]
Sep 15 19:21:50.295992 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:50.295966    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be6a3635-da28-4ad9-b9c8-d8fdfcbcecde" path="/var/lib/kubelet/pods/be6a3635-da28-4ad9-b9c8-d8fdfcbcecde/volumes"
Sep 15 19:21:52.409631 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:21:52.409595    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:22:06.845148 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:06.845072    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7073b8-29824995-jvfc2"]
Sep 15 19:22:06.847512 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:06.847490    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7073b8-29824995-jvfc2"]
Sep 15 19:22:06.867861 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:06.867831    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"]
Sep 15 19:22:06.868077 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:06.868030    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp" podUID="44ab7060-e1b7-42ca-b61b-4f72653a13a8" containerName="payload-processing" containerID="cri-o://4871d5d88feb38cf77a81e70586439d35336f7d1b9c4a4b2f4c80286d0dcd2fc" gracePeriod=30
Sep 15 19:22:06.875728 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:06.875700    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"]
Sep 15 19:22:06.876216 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:06.875918    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6" podUID="de73c1b5-7da7-4c3b-a75e-54c4368a9bc0" containerName="payload-pre-processing" containerID="cri-o://52fcfc6f8379aa6c162e93a875404a1fa667f5253012f0bd170452dcaef9610e" gracePeriod=30
Sep 15 19:22:07.146142 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.146122    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"
Sep 15 19:22:07.149204 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.149182    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"
Sep 15 19:22:07.257599 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.257569    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sr98d\" (UniqueName: \"kubernetes.io/projected/44ab7060-e1b7-42ca-b61b-4f72653a13a8-kube-api-access-sr98d\") pod \"44ab7060-e1b7-42ca-b61b-4f72653a13a8\" (UID: \"44ab7060-e1b7-42ca-b61b-4f72653a13a8\") "
Sep 15 19:22:07.257763 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.257680    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ab7060-e1b7-42ca-b61b-4f72653a13a8-plugins-config-volume\") pod \"44ab7060-e1b7-42ca-b61b-4f72653a13a8\" (UID: \"44ab7060-e1b7-42ca-b61b-4f72653a13a8\") "
Sep 15 19:22:07.257763 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.257701    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sglrn\" (UniqueName: \"kubernetes.io/projected/de73c1b5-7da7-4c3b-a75e-54c4368a9bc0-kube-api-access-sglrn\") pod \"de73c1b5-7da7-4c3b-a75e-54c4368a9bc0\" (UID: \"de73c1b5-7da7-4c3b-a75e-54c4368a9bc0\") "
Sep 15 19:22:07.257763 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.257725    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de73c1b5-7da7-4c3b-a75e-54c4368a9bc0-plugins-config-volume\") pod \"de73c1b5-7da7-4c3b-a75e-54c4368a9bc0\" (UID: \"de73c1b5-7da7-4c3b-a75e-54c4368a9bc0\") "
Sep 15 19:22:07.258115 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.258087    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ab7060-e1b7-42ca-b61b-4f72653a13a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44ab7060-e1b7-42ca-b61b-4f72653a13a8" (UID: "44ab7060-e1b7-42ca-b61b-4f72653a13a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:22:07.258235 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.258154    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de73c1b5-7da7-4c3b-a75e-54c4368a9bc0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de73c1b5-7da7-4c3b-a75e-54c4368a9bc0" (UID: "de73c1b5-7da7-4c3b-a75e-54c4368a9bc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:22:07.259743 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.259722    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ab7060-e1b7-42ca-b61b-4f72653a13a8-kube-api-access-sr98d" (OuterVolumeSpecName: "kube-api-access-sr98d") pod "44ab7060-e1b7-42ca-b61b-4f72653a13a8" (UID: "44ab7060-e1b7-42ca-b61b-4f72653a13a8"). InnerVolumeSpecName "kube-api-access-sr98d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:22:07.259827 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.259776    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de73c1b5-7da7-4c3b-a75e-54c4368a9bc0-kube-api-access-sglrn" (OuterVolumeSpecName: "kube-api-access-sglrn") pod "de73c1b5-7da7-4c3b-a75e-54c4368a9bc0" (UID: "de73c1b5-7da7-4c3b-a75e-54c4368a9bc0"). InnerVolumeSpecName "kube-api-access-sglrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:22:07.332940 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.332912    2550 generic.go:356] "Generic (PLEG): container finished" podID="44ab7060-e1b7-42ca-b61b-4f72653a13a8" containerID="4871d5d88feb38cf77a81e70586439d35336f7d1b9c4a4b2f4c80286d0dcd2fc" exitCode=0
Sep 15 19:22:07.333090 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.332951    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"
Sep 15 19:22:07.333090 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.332981    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp" event={"ID":"44ab7060-e1b7-42ca-b61b-4f72653a13a8","Type":"ContainerDied","Data":"4871d5d88feb38cf77a81e70586439d35336f7d1b9c4a4b2f4c80286d0dcd2fc"}
Sep 15 19:22:07.333090 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.333032    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp" event={"ID":"44ab7060-e1b7-42ca-b61b-4f72653a13a8","Type":"ContainerDied","Data":"00909c39363dcea57211720fabe4b6fc2c018bec4b56cd75391552f45e49197c"}
Sep 15 19:22:07.333090 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.333063    2550 scope.go:117] "RemoveContainer" containerID="4871d5d88feb38cf77a81e70586439d35336f7d1b9c4a4b2f4c80286d0dcd2fc"
Sep 15 19:22:07.334348 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.334296    2550 generic.go:356] "Generic (PLEG): container finished" podID="de73c1b5-7da7-4c3b-a75e-54c4368a9bc0" containerID="52fcfc6f8379aa6c162e93a875404a1fa667f5253012f0bd170452dcaef9610e" exitCode=0
Sep 15 19:22:07.334494 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.334348    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"
Sep 15 19:22:07.334494 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.334349    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6" event={"ID":"de73c1b5-7da7-4c3b-a75e-54c4368a9bc0","Type":"ContainerDied","Data":"52fcfc6f8379aa6c162e93a875404a1fa667f5253012f0bd170452dcaef9610e"}
Sep 15 19:22:07.334494 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.334393    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6" event={"ID":"de73c1b5-7da7-4c3b-a75e-54c4368a9bc0","Type":"ContainerDied","Data":"6a3b9c5298386e046468157b7b743b46f0bdaf3a65b7d819d77f43cdb38f4b02"}
Sep 15 19:22:07.342293 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.342274    2550 scope.go:117] "RemoveContainer" containerID="4871d5d88feb38cf77a81e70586439d35336f7d1b9c4a4b2f4c80286d0dcd2fc"
Sep 15 19:22:07.342632 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:22:07.342609    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4871d5d88feb38cf77a81e70586439d35336f7d1b9c4a4b2f4c80286d0dcd2fc\": container with ID starting with 4871d5d88feb38cf77a81e70586439d35336f7d1b9c4a4b2f4c80286d0dcd2fc not found: ID does not exist" containerID="4871d5d88feb38cf77a81e70586439d35336f7d1b9c4a4b2f4c80286d0dcd2fc"
Sep 15 19:22:07.342695 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.342645    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4871d5d88feb38cf77a81e70586439d35336f7d1b9c4a4b2f4c80286d0dcd2fc"} err="failed to get container status \"4871d5d88feb38cf77a81e70586439d35336f7d1b9c4a4b2f4c80286d0dcd2fc\": rpc error: code = NotFound desc = could not find container \"4871d5d88feb38cf77a81e70586439d35336f7d1b9c4a4b2f4c80286d0dcd2fc\": container with ID starting with 4871d5d88feb38cf77a81e70586439d35336f7d1b9c4a4b2f4c80286d0dcd2fc not found: ID does not exist"
Sep 15 19:22:07.342695 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.342664    2550 scope.go:117] "RemoveContainer" containerID="52fcfc6f8379aa6c162e93a875404a1fa667f5253012f0bd170452dcaef9610e"
Sep 15 19:22:07.350339 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.350322    2550 scope.go:117] "RemoveContainer" containerID="52fcfc6f8379aa6c162e93a875404a1fa667f5253012f0bd170452dcaef9610e"
Sep 15 19:22:07.350583 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:22:07.350567    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52fcfc6f8379aa6c162e93a875404a1fa667f5253012f0bd170452dcaef9610e\": container with ID starting with 52fcfc6f8379aa6c162e93a875404a1fa667f5253012f0bd170452dcaef9610e not found: ID does not exist" containerID="52fcfc6f8379aa6c162e93a875404a1fa667f5253012f0bd170452dcaef9610e"
Sep 15 19:22:07.350644 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.350588    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52fcfc6f8379aa6c162e93a875404a1fa667f5253012f0bd170452dcaef9610e"} err="failed to get container status \"52fcfc6f8379aa6c162e93a875404a1fa667f5253012f0bd170452dcaef9610e\": rpc error: code = NotFound desc = could not find container \"52fcfc6f8379aa6c162e93a875404a1fa667f5253012f0bd170452dcaef9610e\": container with ID starting with 52fcfc6f8379aa6c162e93a875404a1fa667f5253012f0bd170452dcaef9610e not found: ID does not exist"
Sep 15 19:22:07.354584 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.354564    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"]
Sep 15 19:22:07.357663 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.357644    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7073b8-6fbb8b8d7-x2bgp"]
Sep 15 19:22:07.358486 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.358470    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr98d\" (UniqueName: \"kubernetes.io/projected/44ab7060-e1b7-42ca-b61b-4f72653a13a8-kube-api-access-sr98d\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:07.358540 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.358487    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ab7060-e1b7-42ca-b61b-4f72653a13a8-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:07.358540 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.358498    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sglrn\" (UniqueName: \"kubernetes.io/projected/de73c1b5-7da7-4c3b-a75e-54c4368a9bc0-kube-api-access-sglrn\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:07.358540 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.358506    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de73c1b5-7da7-4c3b-a75e-54c4368a9bc0-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:07.366818 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.366797    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"]
Sep 15 19:22:07.370322 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:07.370254    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7073b8-6b6c986694-dzcp6"]
Sep 15 19:22:08.295698 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:08.295669    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44ab7060-e1b7-42ca-b61b-4f72653a13a8" path="/var/lib/kubelet/pods/44ab7060-e1b7-42ca-b61b-4f72653a13a8/volumes"
Sep 15 19:22:08.296065 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:08.296024    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de73c1b5-7da7-4c3b-a75e-54c4368a9bc0" path="/var/lib/kubelet/pods/de73c1b5-7da7-4c3b-a75e-54c4368a9bc0/volumes"
Sep 15 19:22:08.296260 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:08.296247    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e286ffc5-bd97-4023-8482-1aadca768ae6" path="/var/lib/kubelet/pods/e286ffc5-bd97-4023-8482-1aadca768ae6/volumes"
Sep 15 19:22:09.604801 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:09.604765    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:22:24.506817 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:24.506787    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"]
Sep 15 19:22:24.507392 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:24.506992    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64" podUID="b346cff5-cdbd-430d-acbd-571362ef7fea" containerName="payload-processing" containerID="cri-o://9fec8f459796aab49e1900dffe5efebe426c760093319b9cd191429a2a1e4312" gracePeriod=30
Sep 15 19:22:24.766946 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:24.766891    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"
Sep 15 19:22:24.909942 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:24.909907    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b346cff5-cdbd-430d-acbd-571362ef7fea-plugins-config-volume\") pod \"b346cff5-cdbd-430d-acbd-571362ef7fea\" (UID: \"b346cff5-cdbd-430d-acbd-571362ef7fea\") "
Sep 15 19:22:24.910106 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:24.909984    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5tj9\" (UniqueName: \"kubernetes.io/projected/b346cff5-cdbd-430d-acbd-571362ef7fea-kube-api-access-h5tj9\") pod \"b346cff5-cdbd-430d-acbd-571362ef7fea\" (UID: \"b346cff5-cdbd-430d-acbd-571362ef7fea\") "
Sep 15 19:22:24.910364 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:24.910306    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b346cff5-cdbd-430d-acbd-571362ef7fea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b346cff5-cdbd-430d-acbd-571362ef7fea" (UID: "b346cff5-cdbd-430d-acbd-571362ef7fea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:22:24.912067 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:24.912044    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b346cff5-cdbd-430d-acbd-571362ef7fea-kube-api-access-h5tj9" (OuterVolumeSpecName: "kube-api-access-h5tj9") pod "b346cff5-cdbd-430d-acbd-571362ef7fea" (UID: "b346cff5-cdbd-430d-acbd-571362ef7fea"). InnerVolumeSpecName "kube-api-access-h5tj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:22:25.010660 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:25.010634    2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b346cff5-cdbd-430d-acbd-571362ef7fea-plugins-config-volume\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:25.010660 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:25.010658    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5tj9\" (UniqueName: \"kubernetes.io/projected/b346cff5-cdbd-430d-acbd-571362ef7fea-kube-api-access-h5tj9\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:25.427492 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:25.427458    2550 generic.go:356] "Generic (PLEG): container finished" podID="b346cff5-cdbd-430d-acbd-571362ef7fea" containerID="9fec8f459796aab49e1900dffe5efebe426c760093319b9cd191429a2a1e4312" exitCode=0
Sep 15 19:22:25.427644 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:25.427499    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"
Sep 15 19:22:25.427644 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:25.427507    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64" event={"ID":"b346cff5-cdbd-430d-acbd-571362ef7fea","Type":"ContainerDied","Data":"9fec8f459796aab49e1900dffe5efebe426c760093319b9cd191429a2a1e4312"}
Sep 15 19:22:25.427644 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:25.427550    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64" event={"ID":"b346cff5-cdbd-430d-acbd-571362ef7fea","Type":"ContainerDied","Data":"3bedf0d67a74f4a282a07c7c9af59ff8f4f8ad6e6ef0da0884e3c701fbbdbaf5"}
Sep 15 19:22:25.427644 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:25.427571    2550 scope.go:117] "RemoveContainer" containerID="9fec8f459796aab49e1900dffe5efebe426c760093319b9cd191429a2a1e4312"
Sep 15 19:22:25.436788 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:25.436770    2550 scope.go:117] "RemoveContainer" containerID="9fec8f459796aab49e1900dffe5efebe426c760093319b9cd191429a2a1e4312"
Sep 15 19:22:25.437050 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:22:25.437032    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fec8f459796aab49e1900dffe5efebe426c760093319b9cd191429a2a1e4312\": container with ID starting with 9fec8f459796aab49e1900dffe5efebe426c760093319b9cd191429a2a1e4312 not found: ID does not exist" containerID="9fec8f459796aab49e1900dffe5efebe426c760093319b9cd191429a2a1e4312"
Sep 15 19:22:25.437108 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:25.437058    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fec8f459796aab49e1900dffe5efebe426c760093319b9cd191429a2a1e4312"} err="failed to get container status \"9fec8f459796aab49e1900dffe5efebe426c760093319b9cd191429a2a1e4312\": rpc error: code = NotFound desc = could not find container \"9fec8f459796aab49e1900dffe5efebe426c760093319b9cd191429a2a1e4312\": container with ID starting with 9fec8f459796aab49e1900dffe5efebe426c760093319b9cd191429a2a1e4312 not found: ID does not exist"
Sep 15 19:22:25.447629 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:25.447607    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"]
Sep 15 19:22:25.451496 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:25.451475    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-33ff04-7fb47dd7dc-scd64"]
Sep 15 19:22:26.296244 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:26.296211    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b346cff5-cdbd-430d-acbd-571362ef7fea" path="/var/lib/kubelet/pods/b346cff5-cdbd-430d-acbd-571362ef7fea/volumes"
Sep 15 19:22:27.118060 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:27.118028    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:22:31.117331 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:31.117275    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"]
Sep 15 19:22:31.117729 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:31.117543    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l" podUID="9419198c-cfee-455c-8903-2cd4763e2e6e" containerName="istio-proxy" containerID="cri-o://4ec2c7f5ef8233c5d45a1c87ff91d8f87ad45f2d407eccccac891d6dbf05dcd4" gracePeriod=30
Sep 15 19:22:33.372474 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:33.372441    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-tkww4"]
Sep 15 19:22:33.372944 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:33.372619    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6ff85d469c-tkww4" podUID="2666f189-73a4-4051-b0f2-1552f138aec8" containerName="manager" containerID="cri-o://e52c970ba07ed68ca1cbeb7b54e22b21fa48c8352a59751f41b772507288cbde" gracePeriod=30
Sep 15 19:22:33.711974 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:33.711951    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff85d469c-tkww4"
Sep 15 19:22:33.785395 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:33.785358    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2666f189-73a4-4051-b0f2-1552f138aec8-webhook-cert\") pod \"2666f189-73a4-4051-b0f2-1552f138aec8\" (UID: \"2666f189-73a4-4051-b0f2-1552f138aec8\") "
Sep 15 19:22:33.785562 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:33.785433    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2666f189-73a4-4051-b0f2-1552f138aec8-metrics-certs\") pod \"2666f189-73a4-4051-b0f2-1552f138aec8\" (UID: \"2666f189-73a4-4051-b0f2-1552f138aec8\") "
Sep 15 19:22:33.785562 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:33.785514    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkdkw\" (UniqueName: \"kubernetes.io/projected/2666f189-73a4-4051-b0f2-1552f138aec8-kube-api-access-xkdkw\") pod \"2666f189-73a4-4051-b0f2-1552f138aec8\" (UID: \"2666f189-73a4-4051-b0f2-1552f138aec8\") "
Sep 15 19:22:33.787295 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:33.787261    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2666f189-73a4-4051-b0f2-1552f138aec8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2666f189-73a4-4051-b0f2-1552f138aec8" (UID: "2666f189-73a4-4051-b0f2-1552f138aec8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:22:33.787427 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:33.787380    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2666f189-73a4-4051-b0f2-1552f138aec8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2666f189-73a4-4051-b0f2-1552f138aec8" (UID: "2666f189-73a4-4051-b0f2-1552f138aec8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:22:33.787506 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:33.787490    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2666f189-73a4-4051-b0f2-1552f138aec8-kube-api-access-xkdkw" (OuterVolumeSpecName: "kube-api-access-xkdkw") pod "2666f189-73a4-4051-b0f2-1552f138aec8" (UID: "2666f189-73a4-4051-b0f2-1552f138aec8"). InnerVolumeSpecName "kube-api-access-xkdkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:22:33.886215 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:33.886156    2550 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2666f189-73a4-4051-b0f2-1552f138aec8-webhook-cert\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:33.886215 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:33.886177    2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2666f189-73a4-4051-b0f2-1552f138aec8-metrics-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:33.886215 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:33.886186    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkdkw\" (UniqueName: \"kubernetes.io/projected/2666f189-73a4-4051-b0f2-1552f138aec8-kube-api-access-xkdkw\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:34.472484 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:34.472448    2550 generic.go:356] "Generic (PLEG): container finished" podID="2666f189-73a4-4051-b0f2-1552f138aec8" containerID="e52c970ba07ed68ca1cbeb7b54e22b21fa48c8352a59751f41b772507288cbde" exitCode=0
Sep 15 19:22:34.472922 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:34.472488    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff85d469c-tkww4" event={"ID":"2666f189-73a4-4051-b0f2-1552f138aec8","Type":"ContainerDied","Data":"e52c970ba07ed68ca1cbeb7b54e22b21fa48c8352a59751f41b772507288cbde"}
Sep 15 19:22:34.472922 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:34.472497    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff85d469c-tkww4"
Sep 15 19:22:34.472922 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:34.472555    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff85d469c-tkww4" event={"ID":"2666f189-73a4-4051-b0f2-1552f138aec8","Type":"ContainerDied","Data":"0d5e584043611d738268198d1617309c267072272021d690d4a39732bc1bdc2a"}
Sep 15 19:22:34.472922 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:34.472579    2550 scope.go:117] "RemoveContainer" containerID="e52c970ba07ed68ca1cbeb7b54e22b21fa48c8352a59751f41b772507288cbde"
Sep 15 19:22:34.481884 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:34.481865    2550 scope.go:117] "RemoveContainer" containerID="e52c970ba07ed68ca1cbeb7b54e22b21fa48c8352a59751f41b772507288cbde"
Sep 15 19:22:34.482134 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:22:34.482118    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52c970ba07ed68ca1cbeb7b54e22b21fa48c8352a59751f41b772507288cbde\": container with ID starting with e52c970ba07ed68ca1cbeb7b54e22b21fa48c8352a59751f41b772507288cbde not found: ID does not exist" containerID="e52c970ba07ed68ca1cbeb7b54e22b21fa48c8352a59751f41b772507288cbde"
Sep 15 19:22:34.482185 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:34.482141    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52c970ba07ed68ca1cbeb7b54e22b21fa48c8352a59751f41b772507288cbde"} err="failed to get container status \"e52c970ba07ed68ca1cbeb7b54e22b21fa48c8352a59751f41b772507288cbde\": rpc error: code = NotFound desc = could not find container \"e52c970ba07ed68ca1cbeb7b54e22b21fa48c8352a59751f41b772507288cbde\": container with ID starting with e52c970ba07ed68ca1cbeb7b54e22b21fa48c8352a59751f41b772507288cbde not found: ID does not exist"
Sep 15 19:22:34.490092 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:34.490067    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-tkww4"]
Sep 15 19:22:34.493625 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:34.493601    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-tkww4"]
Sep 15 19:22:34.808160 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:34.808124    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:22:36.297333 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.297288    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2666f189-73a4-4051-b0f2-1552f138aec8" path="/var/lib/kubelet/pods/2666f189-73a4-4051-b0f2-1552f138aec8/volumes"
Sep 15 19:22:36.355099 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.355078    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:22:36.406584 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.406516    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9419198c-cfee-455c-8903-2cd4763e2e6e-istiod-ca-cert\") pod \"9419198c-cfee-455c-8903-2cd4763e2e6e\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") "
Sep 15 19:22:36.406584 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.406546    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-podinfo\") pod \"9419198c-cfee-455c-8903-2cd4763e2e6e\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") "
Sep 15 19:22:36.406740 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.406588    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcz6c\" (UniqueName: \"kubernetes.io/projected/9419198c-cfee-455c-8903-2cd4763e2e6e-kube-api-access-vcz6c\") pod \"9419198c-cfee-455c-8903-2cd4763e2e6e\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") "
Sep 15 19:22:36.406740 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.406619    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-envoy\") pod \"9419198c-cfee-455c-8903-2cd4763e2e6e\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") "
Sep 15 19:22:36.406740 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.406656    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-credential-socket\") pod \"9419198c-cfee-455c-8903-2cd4763e2e6e\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") "
Sep 15 19:22:36.406740 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.406721    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-workload-certs\") pod \"9419198c-cfee-455c-8903-2cd4763e2e6e\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") "
Sep 15 19:22:36.406943 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.406749    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-workload-socket\") pod \"9419198c-cfee-455c-8903-2cd4763e2e6e\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") "
Sep 15 19:22:36.406943 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.406796    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-token\") pod \"9419198c-cfee-455c-8903-2cd4763e2e6e\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") "
Sep 15 19:22:36.406943 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.406822    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-data\") pod \"9419198c-cfee-455c-8903-2cd4763e2e6e\" (UID: \"9419198c-cfee-455c-8903-2cd4763e2e6e\") "
Sep 15 19:22:36.407092 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.407005    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9419198c-cfee-455c-8903-2cd4763e2e6e" (UID: "9419198c-cfee-455c-8903-2cd4763e2e6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:22:36.407167 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.407150    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-credential-socket\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:36.407221 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.407007    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9419198c-cfee-455c-8903-2cd4763e2e6e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9419198c-cfee-455c-8903-2cd4763e2e6e" (UID: "9419198c-cfee-455c-8903-2cd4763e2e6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:22:36.407260 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.407215    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9419198c-cfee-455c-8903-2cd4763e2e6e" (UID: "9419198c-cfee-455c-8903-2cd4763e2e6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:22:36.407260 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.407229    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-data" (OuterVolumeSpecName: "istio-data") pod "9419198c-cfee-455c-8903-2cd4763e2e6e" (UID: "9419198c-cfee-455c-8903-2cd4763e2e6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:22:36.407260 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.407158    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9419198c-cfee-455c-8903-2cd4763e2e6e" (UID: "9419198c-cfee-455c-8903-2cd4763e2e6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:22:36.408889 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.408868    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9419198c-cfee-455c-8903-2cd4763e2e6e-kube-api-access-vcz6c" (OuterVolumeSpecName: "kube-api-access-vcz6c") pod "9419198c-cfee-455c-8903-2cd4763e2e6e" (UID: "9419198c-cfee-455c-8903-2cd4763e2e6e"). InnerVolumeSpecName "kube-api-access-vcz6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:22:36.409005 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.408892    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-token" (OuterVolumeSpecName: "istio-token") pod "9419198c-cfee-455c-8903-2cd4763e2e6e" (UID: "9419198c-cfee-455c-8903-2cd4763e2e6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:22:36.409161 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.409145    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9419198c-cfee-455c-8903-2cd4763e2e6e" (UID: "9419198c-cfee-455c-8903-2cd4763e2e6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:22:36.409232 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.409148    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9419198c-cfee-455c-8903-2cd4763e2e6e" (UID: "9419198c-cfee-455c-8903-2cd4763e2e6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:22:36.485004 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.484973    2550 generic.go:356] "Generic (PLEG): container finished" podID="9419198c-cfee-455c-8903-2cd4763e2e6e" containerID="4ec2c7f5ef8233c5d45a1c87ff91d8f87ad45f2d407eccccac891d6dbf05dcd4" exitCode=0
Sep 15 19:22:36.485172 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.485022    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"
Sep 15 19:22:36.485172 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.485036    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l" event={"ID":"9419198c-cfee-455c-8903-2cd4763e2e6e","Type":"ContainerDied","Data":"4ec2c7f5ef8233c5d45a1c87ff91d8f87ad45f2d407eccccac891d6dbf05dcd4"}
Sep 15 19:22:36.485172 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.485079    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l" event={"ID":"9419198c-cfee-455c-8903-2cd4763e2e6e","Type":"ContainerDied","Data":"7aa4fd760514ac86a0eb5414919dcb8f37ee6dfc355a62571819cbf0b4974333"}
Sep 15 19:22:36.485172 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.485100    2550 scope.go:117] "RemoveContainer" containerID="4ec2c7f5ef8233c5d45a1c87ff91d8f87ad45f2d407eccccac891d6dbf05dcd4"
Sep 15 19:22:36.494023 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.494000    2550 scope.go:117] "RemoveContainer" containerID="941622cd9ac6d4d765876b5bb5a2acda89fd852bbc33e33d2cec9a8b5e8132b4"
Sep 15 19:22:36.501674 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.501654    2550 scope.go:117] "RemoveContainer" containerID="4ec2c7f5ef8233c5d45a1c87ff91d8f87ad45f2d407eccccac891d6dbf05dcd4"
Sep 15 19:22:36.501928 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:22:36.501903    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec2c7f5ef8233c5d45a1c87ff91d8f87ad45f2d407eccccac891d6dbf05dcd4\": container with ID starting with 4ec2c7f5ef8233c5d45a1c87ff91d8f87ad45f2d407eccccac891d6dbf05dcd4 not found: ID does not exist" containerID="4ec2c7f5ef8233c5d45a1c87ff91d8f87ad45f2d407eccccac891d6dbf05dcd4"
Sep 15 19:22:36.501974 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.501927    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec2c7f5ef8233c5d45a1c87ff91d8f87ad45f2d407eccccac891d6dbf05dcd4"} err="failed to get container status \"4ec2c7f5ef8233c5d45a1c87ff91d8f87ad45f2d407eccccac891d6dbf05dcd4\": rpc error: code = NotFound desc = could not find container \"4ec2c7f5ef8233c5d45a1c87ff91d8f87ad45f2d407eccccac891d6dbf05dcd4\": container with ID starting with 4ec2c7f5ef8233c5d45a1c87ff91d8f87ad45f2d407eccccac891d6dbf05dcd4 not found: ID does not exist"
Sep 15 19:22:36.501974 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.501942    2550 scope.go:117] "RemoveContainer" containerID="941622cd9ac6d4d765876b5bb5a2acda89fd852bbc33e33d2cec9a8b5e8132b4"
Sep 15 19:22:36.502175 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:22:36.502161    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941622cd9ac6d4d765876b5bb5a2acda89fd852bbc33e33d2cec9a8b5e8132b4\": container with ID starting with 941622cd9ac6d4d765876b5bb5a2acda89fd852bbc33e33d2cec9a8b5e8132b4 not found: ID does not exist" containerID="941622cd9ac6d4d765876b5bb5a2acda89fd852bbc33e33d2cec9a8b5e8132b4"
Sep 15 19:22:36.502211 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.502179    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941622cd9ac6d4d765876b5bb5a2acda89fd852bbc33e33d2cec9a8b5e8132b4"} err="failed to get container status \"941622cd9ac6d4d765876b5bb5a2acda89fd852bbc33e33d2cec9a8b5e8132b4\": rpc error: code = NotFound desc = could not find container \"941622cd9ac6d4d765876b5bb5a2acda89fd852bbc33e33d2cec9a8b5e8132b4\": container with ID starting with 941622cd9ac6d4d765876b5bb5a2acda89fd852bbc33e33d2cec9a8b5e8132b4 not found: ID does not exist"
Sep 15 19:22:36.505645 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.505624    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"]
Sep 15 19:22:36.508261 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.508241    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-token\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:36.508373 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.508264    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-data\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:36.508373 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.508276    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9419198c-cfee-455c-8903-2cd4763e2e6e-istiod-ca-cert\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:36.508373 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.508290    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-podinfo\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:36.508373 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.508303    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcz6c\" (UniqueName: \"kubernetes.io/projected/9419198c-cfee-455c-8903-2cd4763e2e6e-kube-api-access-vcz6c\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:36.508373 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.508354    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-istio-envoy\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:36.508373 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.508367    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-workload-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:36.508579 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.508379    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9419198c-cfee-455c-8903-2cd4763e2e6e-workload-socket\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:22:36.511340 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:36.511306    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-33ff04-openshift-default-8698c8b648-hhk8l"]
Sep 15 19:22:38.295908 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:38.295877    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9419198c-cfee-455c-8903-2cd4763e2e6e" path="/var/lib/kubelet/pods/9419198c-cfee-455c-8903-2cd4763e2e6e/volumes"
Sep 15 19:22:38.414787 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:22:38.414760    2550 scope.go:117] "RemoveContainer" containerID="fd4c45d46bb0a6b439192688ab7e6e5b4bf5dadb896964e83509c345ed04cacd"
Sep 15 19:23:35.302537 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302464    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6ff85d469c-wwntk"]
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302781    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ab7060-e1b7-42ca-b61b-4f72653a13a8" containerName="payload-processing"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302792    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ab7060-e1b7-42ca-b61b-4f72653a13a8" containerName="payload-processing"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302803    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" containerName="istio-proxy"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302810    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" containerName="istio-proxy"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302821    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25ca180-31bf-4e3f-bdfa-08531f231196" containerName="payload-processing"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302827    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25ca180-31bf-4e3f-bdfa-08531f231196" containerName="payload-processing"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302835    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b346cff5-cdbd-430d-acbd-571362ef7fea" containerName="payload-processing"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302841    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b346cff5-cdbd-430d-acbd-571362ef7fea" containerName="payload-processing"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302848    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2666f189-73a4-4051-b0f2-1552f138aec8" containerName="manager"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302853    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2666f189-73a4-4051-b0f2-1552f138aec8" containerName="manager"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302859    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9419198c-cfee-455c-8903-2cd4763e2e6e" containerName="istio-proxy"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302864    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9419198c-cfee-455c-8903-2cd4763e2e6e" containerName="istio-proxy"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302871    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9419198c-cfee-455c-8903-2cd4763e2e6e" containerName="istio-proxy"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302876    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9419198c-cfee-455c-8903-2cd4763e2e6e" containerName="istio-proxy"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302882    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de73c1b5-7da7-4c3b-a75e-54c4368a9bc0" containerName="payload-pre-processing"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302887    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="de73c1b5-7da7-4c3b-a75e-54c4368a9bc0" containerName="payload-pre-processing"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302896    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be6a3635-da28-4ad9-b9c8-d8fdfcbcecde" containerName="payload-pre-processing"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302902    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="be6a3635-da28-4ad9-b9c8-d8fdfcbcecde" containerName="payload-pre-processing"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302949    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2666f189-73a4-4051-b0f2-1552f138aec8" containerName="manager"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302955    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ab7060-e1b7-42ca-b61b-4f72653a13a8" containerName="payload-processing"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302961    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="be6a3635-da28-4ad9-b9c8-d8fdfcbcecde" containerName="payload-pre-processing"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302966    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25ca180-31bf-4e3f-bdfa-08531f231196" containerName="payload-processing"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302974    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9419198c-cfee-455c-8903-2cd4763e2e6e" containerName="istio-proxy"
Sep 15 19:23:35.302971 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302982    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" containerName="istio-proxy"
Sep 15 19:23:35.303748 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302987    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9419198c-cfee-455c-8903-2cd4763e2e6e" containerName="istio-proxy"
Sep 15 19:23:35.303748 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302994    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b346cff5-cdbd-430d-acbd-571362ef7fea" containerName="payload-processing"
Sep 15 19:23:35.303748 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.302999    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" containerName="istio-proxy"
Sep 15 19:23:35.303748 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.303005    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="de73c1b5-7da7-4c3b-a75e-54c4368a9bc0" containerName="payload-pre-processing"
Sep 15 19:23:35.305984 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.305964    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff85d469c-wwntk"
Sep 15 19:23:35.308292 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.308267    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 19:23:35.309084 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.309067    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 19:23:35.309182 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.309165    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 19:23:35.309249 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.309211    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ftvtd\""
Sep 15 19:23:35.321692 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.321668    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-wwntk"]
Sep 15 19:23:35.400577 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.400549    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-metrics-certs\") pod \"maas-controller-6ff85d469c-wwntk\" (UID: \"52d5484f-aab2-4410-8ee5-fa8ad93f40ee\") " pod="opendatahub/maas-controller-6ff85d469c-wwntk"
Sep 15 19:23:35.400577 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.400578    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw7x4\" (UniqueName: \"kubernetes.io/projected/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-kube-api-access-hw7x4\") pod \"maas-controller-6ff85d469c-wwntk\" (UID: \"52d5484f-aab2-4410-8ee5-fa8ad93f40ee\") " pod="opendatahub/maas-controller-6ff85d469c-wwntk"
Sep 15 19:23:35.400766 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.400684    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-webhook-cert\") pod \"maas-controller-6ff85d469c-wwntk\" (UID: \"52d5484f-aab2-4410-8ee5-fa8ad93f40ee\") " pod="opendatahub/maas-controller-6ff85d469c-wwntk"
Sep 15 19:23:35.501439 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.501412    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-webhook-cert\") pod \"maas-controller-6ff85d469c-wwntk\" (UID: \"52d5484f-aab2-4410-8ee5-fa8ad93f40ee\") " pod="opendatahub/maas-controller-6ff85d469c-wwntk"
Sep 15 19:23:35.501579 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.501462    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-metrics-certs\") pod \"maas-controller-6ff85d469c-wwntk\" (UID: \"52d5484f-aab2-4410-8ee5-fa8ad93f40ee\") " pod="opendatahub/maas-controller-6ff85d469c-wwntk"
Sep 15 19:23:35.501579 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.501487    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw7x4\" (UniqueName: \"kubernetes.io/projected/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-kube-api-access-hw7x4\") pod \"maas-controller-6ff85d469c-wwntk\" (UID: \"52d5484f-aab2-4410-8ee5-fa8ad93f40ee\") " pod="opendatahub/maas-controller-6ff85d469c-wwntk"
Sep 15 19:23:35.503779 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.503756    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-metrics-certs\") pod \"maas-controller-6ff85d469c-wwntk\" (UID: \"52d5484f-aab2-4410-8ee5-fa8ad93f40ee\") " pod="opendatahub/maas-controller-6ff85d469c-wwntk"
Sep 15 19:23:35.503897 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.503815    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-webhook-cert\") pod \"maas-controller-6ff85d469c-wwntk\" (UID: \"52d5484f-aab2-4410-8ee5-fa8ad93f40ee\") " pod="opendatahub/maas-controller-6ff85d469c-wwntk"
Sep 15 19:23:35.509166 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.509146    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw7x4\" (UniqueName: \"kubernetes.io/projected/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-kube-api-access-hw7x4\") pod \"maas-controller-6ff85d469c-wwntk\" (UID: \"52d5484f-aab2-4410-8ee5-fa8ad93f40ee\") " pod="opendatahub/maas-controller-6ff85d469c-wwntk"
Sep 15 19:23:35.616495 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.616409    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff85d469c-wwntk"
Sep 15 19:23:35.904997 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.904965    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-wwntk"]
Sep 15 19:23:35.907560 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:23:35.907529    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d5484f_aab2_4410_8ee5_fa8ad93f40ee.slice/crio-56c57c16b998acfca76df483d3593c74ec11249d030370e23f147d7fa957cb63 WatchSource:0}: Error finding container 56c57c16b998acfca76df483d3593c74ec11249d030370e23f147d7fa957cb63: Status 404 returned error can't find the container with id 56c57c16b998acfca76df483d3593c74ec11249d030370e23f147d7fa957cb63
Sep 15 19:23:35.909260 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:35.909245    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:23:36.752135 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:36.752096    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff85d469c-wwntk" event={"ID":"52d5484f-aab2-4410-8ee5-fa8ad93f40ee","Type":"ContainerStarted","Data":"035b09c06cb5304295b65a8a6fdd53d35d082a698fca4bd688f482134892270e"}
Sep 15 19:23:36.752526 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:36.752153    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff85d469c-wwntk" event={"ID":"52d5484f-aab2-4410-8ee5-fa8ad93f40ee","Type":"ContainerStarted","Data":"56c57c16b998acfca76df483d3593c74ec11249d030370e23f147d7fa957cb63"}
Sep 15 19:23:36.752526 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:36.752301    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6ff85d469c-wwntk"
Sep 15 19:23:36.768722 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:36.768680    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6ff85d469c-wwntk" podStartSLOduration=1.418299661 podStartE2EDuration="1.768667732s" totalImagesPullingTime="350.368071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:23:36.766735113 +0000 UTC m=+1979.043503970" watchObservedRunningTime="2026-09-15 19:23:36.768667732 +0000 UTC m=+1979.045436588"
Sep 15 19:23:47.760085 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:47.760051    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6ff85d469c-wwntk"
Sep 15 19:23:58.113101 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:58.113068    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:23:59.256488 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.256443    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-wwntk"]
Sep 15 19:23:59.256956 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.256652    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6ff85d469c-wwntk" podUID="52d5484f-aab2-4410-8ee5-fa8ad93f40ee" containerName="manager" containerID="cri-o://035b09c06cb5304295b65a8a6fdd53d35d082a698fca4bd688f482134892270e" gracePeriod=30
Sep 15 19:23:59.490419 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.490379    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff85d469c-wwntk"
Sep 15 19:23:59.593913 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.593877    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-webhook-cert\") pod \"52d5484f-aab2-4410-8ee5-fa8ad93f40ee\" (UID: \"52d5484f-aab2-4410-8ee5-fa8ad93f40ee\") "
Sep 15 19:23:59.594094 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.593943    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-metrics-certs\") pod \"52d5484f-aab2-4410-8ee5-fa8ad93f40ee\" (UID: \"52d5484f-aab2-4410-8ee5-fa8ad93f40ee\") "
Sep 15 19:23:59.594094 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.594038    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw7x4\" (UniqueName: \"kubernetes.io/projected/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-kube-api-access-hw7x4\") pod \"52d5484f-aab2-4410-8ee5-fa8ad93f40ee\" (UID: \"52d5484f-aab2-4410-8ee5-fa8ad93f40ee\") "
Sep 15 19:23:59.596011 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.595976    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-kube-api-access-hw7x4" (OuterVolumeSpecName: "kube-api-access-hw7x4") pod "52d5484f-aab2-4410-8ee5-fa8ad93f40ee" (UID: "52d5484f-aab2-4410-8ee5-fa8ad93f40ee"). InnerVolumeSpecName "kube-api-access-hw7x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:23:59.596130 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.596019    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "52d5484f-aab2-4410-8ee5-fa8ad93f40ee" (UID: "52d5484f-aab2-4410-8ee5-fa8ad93f40ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:23:59.596130 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.596043    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "52d5484f-aab2-4410-8ee5-fa8ad93f40ee" (UID: "52d5484f-aab2-4410-8ee5-fa8ad93f40ee"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:23:59.695160 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.695132    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw7x4\" (UniqueName: \"kubernetes.io/projected/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-kube-api-access-hw7x4\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:23:59.695160 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.695158    2550 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-webhook-cert\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:23:59.695332 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.695167    2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d5484f-aab2-4410-8ee5-fa8ad93f40ee-metrics-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:23:59.854932 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.854862    2550 generic.go:356] "Generic (PLEG): container finished" podID="52d5484f-aab2-4410-8ee5-fa8ad93f40ee" containerID="035b09c06cb5304295b65a8a6fdd53d35d082a698fca4bd688f482134892270e" exitCode=0
Sep 15 19:23:59.854932 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.854901    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff85d469c-wwntk" event={"ID":"52d5484f-aab2-4410-8ee5-fa8ad93f40ee","Type":"ContainerDied","Data":"035b09c06cb5304295b65a8a6fdd53d35d082a698fca4bd688f482134892270e"}
Sep 15 19:23:59.855093 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.854941    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff85d469c-wwntk" event={"ID":"52d5484f-aab2-4410-8ee5-fa8ad93f40ee","Type":"ContainerDied","Data":"56c57c16b998acfca76df483d3593c74ec11249d030370e23f147d7fa957cb63"}
Sep 15 19:23:59.855093 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.854963    2550 scope.go:117] "RemoveContainer" containerID="035b09c06cb5304295b65a8a6fdd53d35d082a698fca4bd688f482134892270e"
Sep 15 19:23:59.855093 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.854904    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff85d469c-wwntk"
Sep 15 19:23:59.863506 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.863487    2550 scope.go:117] "RemoveContainer" containerID="035b09c06cb5304295b65a8a6fdd53d35d082a698fca4bd688f482134892270e"
Sep 15 19:23:59.863789 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:23:59.863770    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035b09c06cb5304295b65a8a6fdd53d35d082a698fca4bd688f482134892270e\": container with ID starting with 035b09c06cb5304295b65a8a6fdd53d35d082a698fca4bd688f482134892270e not found: ID does not exist" containerID="035b09c06cb5304295b65a8a6fdd53d35d082a698fca4bd688f482134892270e"
Sep 15 19:23:59.863839 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.863798    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035b09c06cb5304295b65a8a6fdd53d35d082a698fca4bd688f482134892270e"} err="failed to get container status \"035b09c06cb5304295b65a8a6fdd53d35d082a698fca4bd688f482134892270e\": rpc error: code = NotFound desc = could not find container \"035b09c06cb5304295b65a8a6fdd53d35d082a698fca4bd688f482134892270e\": container with ID starting with 035b09c06cb5304295b65a8a6fdd53d35d082a698fca4bd688f482134892270e not found: ID does not exist"
Sep 15 19:23:59.876982 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.876960    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-wwntk"]
Sep 15 19:23:59.880085 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:23:59.880063    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-wwntk"]
Sep 15 19:24:00.295135 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.295104    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52d5484f-aab2-4410-8ee5-fa8ad93f40ee" path="/var/lib/kubelet/pods/52d5484f-aab2-4410-8ee5-fa8ad93f40ee/volumes"
Sep 15 19:24:00.663137 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.663063    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6ff85d469c-zlv9v"]
Sep 15 19:24:00.663413 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.663401    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52d5484f-aab2-4410-8ee5-fa8ad93f40ee" containerName="manager"
Sep 15 19:24:00.663465 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.663415    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="52d5484f-aab2-4410-8ee5-fa8ad93f40ee" containerName="manager"
Sep 15 19:24:00.663465 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.663427    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" containerName="istio-proxy"
Sep 15 19:24:00.663465 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.663433    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9949d4e0-0c2a-4c80-8e76-bd770ce6444c" containerName="istio-proxy"
Sep 15 19:24:00.663738 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.663488    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="52d5484f-aab2-4410-8ee5-fa8ad93f40ee" containerName="manager"
Sep 15 19:24:00.667842 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.667820    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff85d469c-zlv9v"
Sep 15 19:24:00.670894 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.670870    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ftvtd\""
Sep 15 19:24:00.670894 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.670885    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 19:24:00.671076 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.670872    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 19:24:00.671076 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.670926    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 19:24:00.675730 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.675703    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-zlv9v"]
Sep 15 19:24:00.803421 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.803387    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/09b6f142-4e28-48c4-9774-67672c14415d-webhook-cert\") pod \"maas-controller-6ff85d469c-zlv9v\" (UID: \"09b6f142-4e28-48c4-9774-67672c14415d\") " pod="opendatahub/maas-controller-6ff85d469c-zlv9v"
Sep 15 19:24:00.803589 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.803430    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09b6f142-4e28-48c4-9774-67672c14415d-metrics-certs\") pod \"maas-controller-6ff85d469c-zlv9v\" (UID: \"09b6f142-4e28-48c4-9774-67672c14415d\") " pod="opendatahub/maas-controller-6ff85d469c-zlv9v"
Sep 15 19:24:00.803589 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.803515    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w6nn\" (UniqueName: \"kubernetes.io/projected/09b6f142-4e28-48c4-9774-67672c14415d-kube-api-access-7w6nn\") pod \"maas-controller-6ff85d469c-zlv9v\" (UID: \"09b6f142-4e28-48c4-9774-67672c14415d\") " pod="opendatahub/maas-controller-6ff85d469c-zlv9v"
Sep 15 19:24:00.904167 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.904136    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w6nn\" (UniqueName: \"kubernetes.io/projected/09b6f142-4e28-48c4-9774-67672c14415d-kube-api-access-7w6nn\") pod \"maas-controller-6ff85d469c-zlv9v\" (UID: \"09b6f142-4e28-48c4-9774-67672c14415d\") " pod="opendatahub/maas-controller-6ff85d469c-zlv9v"
Sep 15 19:24:00.904340 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.904197    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/09b6f142-4e28-48c4-9774-67672c14415d-webhook-cert\") pod \"maas-controller-6ff85d469c-zlv9v\" (UID: \"09b6f142-4e28-48c4-9774-67672c14415d\") " pod="opendatahub/maas-controller-6ff85d469c-zlv9v"
Sep 15 19:24:00.904340 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.904219    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09b6f142-4e28-48c4-9774-67672c14415d-metrics-certs\") pod \"maas-controller-6ff85d469c-zlv9v\" (UID: \"09b6f142-4e28-48c4-9774-67672c14415d\") " pod="opendatahub/maas-controller-6ff85d469c-zlv9v"
Sep 15 19:24:00.906754 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.906722    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09b6f142-4e28-48c4-9774-67672c14415d-metrics-certs\") pod \"maas-controller-6ff85d469c-zlv9v\" (UID: \"09b6f142-4e28-48c4-9774-67672c14415d\") " pod="opendatahub/maas-controller-6ff85d469c-zlv9v"
Sep 15 19:24:00.906859 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.906722    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/09b6f142-4e28-48c4-9774-67672c14415d-webhook-cert\") pod \"maas-controller-6ff85d469c-zlv9v\" (UID: \"09b6f142-4e28-48c4-9774-67672c14415d\") " pod="opendatahub/maas-controller-6ff85d469c-zlv9v"
Sep 15 19:24:00.911107 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.911089    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w6nn\" (UniqueName: \"kubernetes.io/projected/09b6f142-4e28-48c4-9774-67672c14415d-kube-api-access-7w6nn\") pod \"maas-controller-6ff85d469c-zlv9v\" (UID: \"09b6f142-4e28-48c4-9774-67672c14415d\") " pod="opendatahub/maas-controller-6ff85d469c-zlv9v"
Sep 15 19:24:00.979694 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:00.979617    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff85d469c-zlv9v"
Sep 15 19:24:01.247574 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:01.247496    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ff85d469c-zlv9v"]
Sep 15 19:24:01.250827 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:24:01.250805    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b6f142_4e28_48c4_9774_67672c14415d.slice/crio-c0e9c85017f02db548b3eea6372748ac02a41b975a9da88136153fd4999303cf WatchSource:0}: Error finding container c0e9c85017f02db548b3eea6372748ac02a41b975a9da88136153fd4999303cf: Status 404 returned error can't find the container with id c0e9c85017f02db548b3eea6372748ac02a41b975a9da88136153fd4999303cf
Sep 15 19:24:01.871996 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:01.871958    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff85d469c-zlv9v" event={"ID":"09b6f142-4e28-48c4-9774-67672c14415d","Type":"ContainerStarted","Data":"734d4b8eea9c7b950144c7b9dae7f8254511ddef6691e508c88fcf019e41b468"}
Sep 15 19:24:01.872392 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:01.872018    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff85d469c-zlv9v" event={"ID":"09b6f142-4e28-48c4-9774-67672c14415d","Type":"ContainerStarted","Data":"c0e9c85017f02db548b3eea6372748ac02a41b975a9da88136153fd4999303cf"}
Sep 15 19:24:01.872392 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:01.872091    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6ff85d469c-zlv9v"
Sep 15 19:24:01.888108 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:01.888065    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6ff85d469c-zlv9v" podStartSLOduration=1.456673796 podStartE2EDuration="1.888052156s" totalImagesPullingTime="431.37836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:24:01.887229487 +0000 UTC m=+2004.163998344" watchObservedRunningTime="2026-09-15 19:24:01.888052156 +0000 UTC m=+2004.164821012"
Sep 15 19:24:12.878564 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:12.878534    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6ff85d469c-zlv9v"
Sep 15 19:24:22.704823 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:22.704784    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:24:30.409241 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:30.409201    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:24:38.705292 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:38.705258    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:24:57.319732 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:24:57.319700    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:25:04.017631 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:25:04.017559    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:26:34.107069 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:26:34.107031    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:26:41.912015 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:26:41.911979    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:26:54.804707 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:26:54.804660    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:27:11.610558 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:11.610517    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:27:26.404808 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:26.404766    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:27:34.507551 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:34.507512    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:27:41.413430 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:41.413390    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:27:49.106707 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.106673    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:27:49.580833 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.580797    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"]
Sep 15 19:27:49.584589 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.584568    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:49.593976 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.593948    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"]
Sep 15 19:27:49.627340 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.627294    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39533bac-a6f1-4c71-9cb7-e0cf68c1542e-maas-api-tls\") pod \"maas-api-86d9ddc875-s9gxs\" (UID: \"39533bac-a6f1-4c71-9cb7-e0cf68c1542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:49.627456 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.627358    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnlwd\" (UniqueName: \"kubernetes.io/projected/39533bac-a6f1-4c71-9cb7-e0cf68c1542e-kube-api-access-vnlwd\") pod \"maas-api-86d9ddc875-s9gxs\" (UID: \"39533bac-a6f1-4c71-9cb7-e0cf68c1542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:49.627456 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.627404    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39533bac-a6f1-4c71-9cb7-e0cf68c1542e-openshift-service-ca\") pod \"maas-api-86d9ddc875-s9gxs\" (UID: \"39533bac-a6f1-4c71-9cb7-e0cf68c1542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:49.627456 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.627428    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39533bac-a6f1-4c71-9cb7-e0cf68c1542e-metrics-certs\") pod \"maas-api-86d9ddc875-s9gxs\" (UID: \"39533bac-a6f1-4c71-9cb7-e0cf68c1542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:49.728702 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.728660    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39533bac-a6f1-4c71-9cb7-e0cf68c1542e-maas-api-tls\") pod \"maas-api-86d9ddc875-s9gxs\" (UID: \"39533bac-a6f1-4c71-9cb7-e0cf68c1542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:49.728866 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.728717    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnlwd\" (UniqueName: \"kubernetes.io/projected/39533bac-a6f1-4c71-9cb7-e0cf68c1542e-kube-api-access-vnlwd\") pod \"maas-api-86d9ddc875-s9gxs\" (UID: \"39533bac-a6f1-4c71-9cb7-e0cf68c1542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:49.728866 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.728769    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39533bac-a6f1-4c71-9cb7-e0cf68c1542e-openshift-service-ca\") pod \"maas-api-86d9ddc875-s9gxs\" (UID: \"39533bac-a6f1-4c71-9cb7-e0cf68c1542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:49.728866 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.728796    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39533bac-a6f1-4c71-9cb7-e0cf68c1542e-metrics-certs\") pod \"maas-api-86d9ddc875-s9gxs\" (UID: \"39533bac-a6f1-4c71-9cb7-e0cf68c1542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:49.729530 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.729504    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39533bac-a6f1-4c71-9cb7-e0cf68c1542e-openshift-service-ca\") pod \"maas-api-86d9ddc875-s9gxs\" (UID: \"39533bac-a6f1-4c71-9cb7-e0cf68c1542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:49.731304 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.731280    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39533bac-a6f1-4c71-9cb7-e0cf68c1542e-metrics-certs\") pod \"maas-api-86d9ddc875-s9gxs\" (UID: \"39533bac-a6f1-4c71-9cb7-e0cf68c1542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:49.731407 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.731342    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39533bac-a6f1-4c71-9cb7-e0cf68c1542e-maas-api-tls\") pod \"maas-api-86d9ddc875-s9gxs\" (UID: \"39533bac-a6f1-4c71-9cb7-e0cf68c1542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:49.736638 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.736610    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnlwd\" (UniqueName: \"kubernetes.io/projected/39533bac-a6f1-4c71-9cb7-e0cf68c1542e-kube-api-access-vnlwd\") pod \"maas-api-86d9ddc875-s9gxs\" (UID: \"39533bac-a6f1-4c71-9cb7-e0cf68c1542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:49.895212 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:49.895131    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:50.167375 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:50.167348    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"]
Sep 15 19:27:50.169668 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:27:50.169634    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39533bac_a6f1_4c71_9cb7_e0cf68c1542e.slice/crio-086c61e5f3bc3786fce1ff984fc9b43125fd4e63e449ddc1db147a9e31aa048d WatchSource:0}: Error finding container 086c61e5f3bc3786fce1ff984fc9b43125fd4e63e449ddc1db147a9e31aa048d: Status 404 returned error can't find the container with id 086c61e5f3bc3786fce1ff984fc9b43125fd4e63e449ddc1db147a9e31aa048d
Sep 15 19:27:50.900672 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:50.900592    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs" event={"ID":"39533bac-a6f1-4c71-9cb7-e0cf68c1542e","Type":"ContainerStarted","Data":"b3d92d5c151b225317a0061ec6089699a6e3ffbc72ed31c8752b9c9257638cb6"}
Sep 15 19:27:50.900672 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:50.900645    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs" event={"ID":"39533bac-a6f1-4c71-9cb7-e0cf68c1542e","Type":"ContainerStarted","Data":"086c61e5f3bc3786fce1ff984fc9b43125fd4e63e449ddc1db147a9e31aa048d"}
Sep 15 19:27:50.900672 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:50.900671    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:50.918599 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:50.918555    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs" podStartSLOduration=1.555695346 podStartE2EDuration="1.918540626s" totalImagesPullingTime="362.84528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:27:50.916830011 +0000 UTC m=+2233.193598892" watchObservedRunningTime="2026-09-15 19:27:50.918540626 +0000 UTC m=+2233.195309482"
Sep 15 19:27:56.209864 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:56.209828    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:27:56.911508 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:56.911475    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-s9gxs"
Sep 15 19:27:56.965993 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:56.965958    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"]
Sep 15 19:27:56.966207 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:56.966166    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f" podUID="0afbf017-6c7f-45f6-9017-2ec92176ca0f" containerName="maas-api" containerID="cri-o://c6041f0426d70cdbd3e2276c3907f9a5c3cfb118125e915569b9cf50242d59cb" gracePeriod=30
Sep 15 19:27:57.734933 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:57.734889    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f" podUID="0afbf017-6c7f-45f6-9017-2ec92176ca0f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.35:8443/health\": dial tcp 10.133.0.35:8443: connect: connection refused"
Sep 15 19:27:58.211962 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.211931    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:27:58.302545 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.302520    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0afbf017-6c7f-45f6-9017-2ec92176ca0f-openshift-service-ca\") pod \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") "
Sep 15 19:27:58.302702 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.302579    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxj2g\" (UniqueName: \"kubernetes.io/projected/0afbf017-6c7f-45f6-9017-2ec92176ca0f-kube-api-access-gxj2g\") pod \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") "
Sep 15 19:27:58.302702 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.302648    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afbf017-6c7f-45f6-9017-2ec92176ca0f-metrics-certs\") pod \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") "
Sep 15 19:27:58.302702 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.302697    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0afbf017-6c7f-45f6-9017-2ec92176ca0f-maas-api-tls\") pod \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\" (UID: \"0afbf017-6c7f-45f6-9017-2ec92176ca0f\") "
Sep 15 19:27:58.302872 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.302844    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0afbf017-6c7f-45f6-9017-2ec92176ca0f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0afbf017-6c7f-45f6-9017-2ec92176ca0f" (UID: "0afbf017-6c7f-45f6-9017-2ec92176ca0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:27:58.302972 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.302952    2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0afbf017-6c7f-45f6-9017-2ec92176ca0f-openshift-service-ca\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:27:58.304702 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.304675    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0afbf017-6c7f-45f6-9017-2ec92176ca0f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0afbf017-6c7f-45f6-9017-2ec92176ca0f" (UID: "0afbf017-6c7f-45f6-9017-2ec92176ca0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:27:58.304821 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.304703    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0afbf017-6c7f-45f6-9017-2ec92176ca0f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0afbf017-6c7f-45f6-9017-2ec92176ca0f" (UID: "0afbf017-6c7f-45f6-9017-2ec92176ca0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:27:58.304821 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.304798    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0afbf017-6c7f-45f6-9017-2ec92176ca0f-kube-api-access-gxj2g" (OuterVolumeSpecName: "kube-api-access-gxj2g") pod "0afbf017-6c7f-45f6-9017-2ec92176ca0f" (UID: "0afbf017-6c7f-45f6-9017-2ec92176ca0f"). InnerVolumeSpecName "kube-api-access-gxj2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:27:58.404098 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.404073    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0afbf017-6c7f-45f6-9017-2ec92176ca0f-maas-api-tls\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:27:58.404098 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.404094    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxj2g\" (UniqueName: \"kubernetes.io/projected/0afbf017-6c7f-45f6-9017-2ec92176ca0f-kube-api-access-gxj2g\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:27:58.404237 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.404104    2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afbf017-6c7f-45f6-9017-2ec92176ca0f-metrics-certs\") on node \"ip-10-0-129-227.ec2.internal\" DevicePath \"\""
Sep 15 19:27:58.939352 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.939300    2550 generic.go:356] "Generic (PLEG): container finished" podID="0afbf017-6c7f-45f6-9017-2ec92176ca0f" containerID="c6041f0426d70cdbd3e2276c3907f9a5c3cfb118125e915569b9cf50242d59cb" exitCode=0
Sep 15 19:27:58.939796 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.939361    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"
Sep 15 19:27:58.939796 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.939378    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f" event={"ID":"0afbf017-6c7f-45f6-9017-2ec92176ca0f","Type":"ContainerDied","Data":"c6041f0426d70cdbd3e2276c3907f9a5c3cfb118125e915569b9cf50242d59cb"}
Sep 15 19:27:58.939796 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.939420    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f" event={"ID":"0afbf017-6c7f-45f6-9017-2ec92176ca0f","Type":"ContainerDied","Data":"4d71a01e2961890a48517ad6e7e8a581ed0ddbd0a95842ad45e05eda81e995bd"}
Sep 15 19:27:58.939796 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.939441    2550 scope.go:117] "RemoveContainer" containerID="c6041f0426d70cdbd3e2276c3907f9a5c3cfb118125e915569b9cf50242d59cb"
Sep 15 19:27:58.948811 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.948795    2550 scope.go:117] "RemoveContainer" containerID="c6041f0426d70cdbd3e2276c3907f9a5c3cfb118125e915569b9cf50242d59cb"
Sep 15 19:27:58.949059 ip-10-0-129-227 kubenswrapper[2550]: E0915 19:27:58.949041    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6041f0426d70cdbd3e2276c3907f9a5c3cfb118125e915569b9cf50242d59cb\": container with ID starting with c6041f0426d70cdbd3e2276c3907f9a5c3cfb118125e915569b9cf50242d59cb not found: ID does not exist" containerID="c6041f0426d70cdbd3e2276c3907f9a5c3cfb118125e915569b9cf50242d59cb"
Sep 15 19:27:58.949129 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.949066    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6041f0426d70cdbd3e2276c3907f9a5c3cfb118125e915569b9cf50242d59cb"} err="failed to get container status \"c6041f0426d70cdbd3e2276c3907f9a5c3cfb118125e915569b9cf50242d59cb\": rpc error: code = NotFound desc = could not find container \"c6041f0426d70cdbd3e2276c3907f9a5c3cfb118125e915569b9cf50242d59cb\": container with ID starting with c6041f0426d70cdbd3e2276c3907f9a5c3cfb118125e915569b9cf50242d59cb not found: ID does not exist"
Sep 15 19:27:58.961081 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.961057    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"]
Sep 15 19:27:58.962951 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:27:58.962930    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-f8b6f"]
Sep 15 19:28:00.295287 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:00.295255    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0afbf017-6c7f-45f6-9017-2ec92176ca0f" path="/var/lib/kubelet/pods/0afbf017-6c7f-45f6-9017-2ec92176ca0f/volumes"
Sep 15 19:28:02.003978 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:02.003943    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:28:09.040862 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:09.040822    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sgjbh"]
Sep 15 19:28:14.664402 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:14.664370    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vgmvv_f05cb8d3-8b0d-45c8-aeb2-a7ecb44a08ae/global-pull-secret-syncer/0.log"
Sep 15 19:28:14.750373 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:14.750344    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fnczv_b3707c81-43dd-4142-9c7b-82b12577c423/konnectivity-agent/0.log"
Sep 15 19:28:14.818586 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:14.818557    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-227.ec2.internal_527678610c0f9e9b262ca08d3e160568/haproxy/0.log"
Sep 15 19:28:19.401054 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:19.401013    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55c58d9c44-ddcsp_e8cb3dc4-59a9-4174-ae0e-bb38a3088dbc/authorino/0.log"
Sep 15 19:28:19.430085 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:19.430053    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-fmrzn_8f3b364f-b9f7-4642-91a5-0da94194e3b2/manager/0.log"
Sep 15 19:28:19.543860 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:19.543831    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-sgjbh_cfe018cd-6e09-4ae5-9a76-1677382ff925/limitador/0.log"
Sep 15 19:28:19.626094 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:19.626066    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-878xj_05f61967-56f3-43eb-a1b9-891c7bb05b6d/manager/0.log"
Sep 15 19:28:21.692723 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:21.692688    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rqcwz_0dfcf813-4f23-49da-a941-71fecec82235/node-exporter/0.log"
Sep 15 19:28:21.708286 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:21.708265    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rqcwz_0dfcf813-4f23-49da-a941-71fecec82235/kube-rbac-proxy/0.log"
Sep 15 19:28:21.727068 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:21.727044    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rqcwz_0dfcf813-4f23-49da-a941-71fecec82235/init-textfile/0.log"
Sep 15 19:28:24.966210 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:24.966175    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"]
Sep 15 19:28:24.966825 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:24.966802    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0afbf017-6c7f-45f6-9017-2ec92176ca0f" containerName="maas-api"
Sep 15 19:28:24.966910 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:24.966828    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0afbf017-6c7f-45f6-9017-2ec92176ca0f" containerName="maas-api"
Sep 15 19:28:24.966965 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:24.966918    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0afbf017-6c7f-45f6-9017-2ec92176ca0f" containerName="maas-api"
Sep 15 19:28:24.970018 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:24.969997    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:24.972418 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:24.972399    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7s247\"/\"openshift-service-ca.crt\""
Sep 15 19:28:24.972516 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:24.972498    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7s247\"/\"kube-root-ca.crt\""
Sep 15 19:28:24.973168 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:24.973151    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7s247\"/\"default-dockercfg-jg7th\""
Sep 15 19:28:24.980323 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:24.980295    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"]
Sep 15 19:28:25.119846 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.119807    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40-lib-modules\") pod \"perf-node-gather-daemonset-4ndlz\" (UID: \"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.119846 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.119850    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8skld\" (UniqueName: \"kubernetes.io/projected/69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40-kube-api-access-8skld\") pod \"perf-node-gather-daemonset-4ndlz\" (UID: \"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.120034 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.119885    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40-proc\") pod \"perf-node-gather-daemonset-4ndlz\" (UID: \"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.120034 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.119912    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40-sys\") pod \"perf-node-gather-daemonset-4ndlz\" (UID: \"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.120034 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.119987    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40-podres\") pod \"perf-node-gather-daemonset-4ndlz\" (UID: \"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.221200 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.221128    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40-podres\") pod \"perf-node-gather-daemonset-4ndlz\" (UID: \"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.221200 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.221175    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40-lib-modules\") pod \"perf-node-gather-daemonset-4ndlz\" (UID: \"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.221441 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.221205    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skld\" (UniqueName: \"kubernetes.io/projected/69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40-kube-api-access-8skld\") pod \"perf-node-gather-daemonset-4ndlz\" (UID: \"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.221441 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.221245    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40-proc\") pod \"perf-node-gather-daemonset-4ndlz\" (UID: \"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.221441 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.221276    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40-sys\") pod \"perf-node-gather-daemonset-4ndlz\" (UID: \"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.221441 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.221281    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40-podres\") pod \"perf-node-gather-daemonset-4ndlz\" (UID: \"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.221441 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.221301    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40-lib-modules\") pod \"perf-node-gather-daemonset-4ndlz\" (UID: \"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.221441 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.221356    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40-proc\") pod \"perf-node-gather-daemonset-4ndlz\" (UID: \"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.221441 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.221364    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40-sys\") pod \"perf-node-gather-daemonset-4ndlz\" (UID: \"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.229645 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.229624    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skld\" (UniqueName: \"kubernetes.io/projected/69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40-kube-api-access-8skld\") pod \"perf-node-gather-daemonset-4ndlz\" (UID: \"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.280307 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.280285    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:25.545715 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:25.545684    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"]
Sep 15 19:28:25.548083 ip-10-0-129-227 kubenswrapper[2550]: W0915 19:28:25.548040    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod69e24d46_8d57_4fc3_8b4c_9b0f5bfadd40.slice/crio-5e55216a2ac5b45872bab99fd86fe70655e76ca5a890a11d856cc376fd711970 WatchSource:0}: Error finding container 5e55216a2ac5b45872bab99fd86fe70655e76ca5a890a11d856cc376fd711970: Status 404 returned error can't find the container with id 5e55216a2ac5b45872bab99fd86fe70655e76ca5a890a11d856cc376fd711970
Sep 15 19:28:26.066232 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:26.066200    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz" event={"ID":"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40","Type":"ContainerStarted","Data":"75401c0c5900b41f3354db5064e03af1fdd82e372bdb696c85212750c7bf8423"}
Sep 15 19:28:26.066598 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:26.066260    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz" event={"ID":"69e24d46-8d57-4fc3-8b4c-9b0f5bfadd40","Type":"ContainerStarted","Data":"5e55216a2ac5b45872bab99fd86fe70655e76ca5a890a11d856cc376fd711970"}
Sep 15 19:28:26.066598 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:26.066292    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:26.082150 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:26.082107    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz" podStartSLOduration=2.082092209 podStartE2EDuration="2.082092209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 19:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:28:26.08053376 +0000 UTC m=+2268.357302616" watchObservedRunningTime="2026-09-15 19:28:26.082092209 +0000 UTC m=+2268.358861064"
Sep 15 19:28:26.942771 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:26.942741    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bq2c6_69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31/dns/0.log"
Sep 15 19:28:26.961987 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:26.961953    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bq2c6_69a08e4e-d5c2-4e7d-bfd8-61e8f8325b31/kube-rbac-proxy/0.log"
Sep 15 19:28:27.086861 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:27.086817    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5n5wz_9fd5a36a-c351-4e64-8b75-abf7e06279a0/dns-node-resolver/0.log"
Sep 15 19:28:27.664066 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:27.664031    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55b68bbff6-rd86t_a6c35301-640a-40d1-bafe-7fb5acbc911b/registry/0.log"
Sep 15 19:28:27.684263 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:27.684235    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-776nl_45a116a2-6657-4785-8769-056a5708db32/node-ca/0.log"
Sep 15 19:28:29.062710 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:29.062681    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b76b67bf4-tsmb8_cec8a25b-eef6-4414-afbf-849f642211f4/kube-auth-proxy/0.log"
Sep 15 19:28:29.117490 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:29.117458    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-ht96l_8190ca1e-7e85-490e-b665-4712192e4d60/istio-proxy/0.log"
Sep 15 19:28:29.121330 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:29.121287    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-ht96l_8190ca1e-7e85-490e-b665-4712192e4d60/istio-proxy/1.log"
Sep 15 19:28:30.083559 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:30.083530    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lvxks_8216bdee-a7c9-4523-8732-ba0dbba092fd/serve-healthcheck-canary/0.log"
Sep 15 19:28:30.652017 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:30.651985    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qz4bv_8ca2cbaa-a242-425a-8886-c327cee392b2/kube-rbac-proxy/0.log"
Sep 15 19:28:30.670705 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:30.670682    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qz4bv_8ca2cbaa-a242-425a-8886-c327cee392b2/exporter/0.log"
Sep 15 19:28:30.691266 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:30.691243    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qz4bv_8ca2cbaa-a242-425a-8886-c327cee392b2/extractor/0.log"
Sep 15 19:28:32.080409 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:32.080383    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7s247/perf-node-gather-daemonset-4ndlz"
Sep 15 19:28:32.908601 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:32.908572    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-x8g9g_85a2fbc0-1096-45fc-ac95-95f70dca3516/manager/0.log"
Sep 15 19:28:33.278825 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:33.278796    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zg5g5_a0002c4e-6120-4e46-bb2c-2362d920e0e3/manager/0.log"
Sep 15 19:28:33.349110 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:33.349079    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6ff85d469c-zlv9v_09b6f142-4e28-48c4-9774-67672c14415d/manager/0.log"
Sep 15 19:28:33.373115 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:33.373090    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ncpmd_a817227a-1d88-42ea-8e62-37bbadb82cf7/server/0.log"
Sep 15 19:28:33.684768 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:33.684737    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7467f6b4cf-z5sbq_1fc2947b-2cbe-4f61-8b52-7231498ceac3/manager/0.log"
Sep 15 19:28:33.706167 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:33.706146    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7467f6b4cf-z5sbq_1fc2947b-2cbe-4f61-8b52-7231498ceac3/copy-manifests/0.log"
Sep 15 19:28:35.792503 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:35.792478    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b76b67bf4-tsmb8_cec8a25b-eef6-4414-afbf-849f642211f4/kube-auth-proxy/0.log"
Sep 15 19:28:35.833624 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:35.833594    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-ht96l_8190ca1e-7e85-490e-b665-4712192e4d60/istio-proxy/0.log"
Sep 15 19:28:35.837515 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:35.837492    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-ht96l_8190ca1e-7e85-490e-b665-4712192e4d60/istio-proxy/1.log"
Sep 15 19:28:37.731472 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:37.731428    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b76b67bf4-tsmb8_cec8a25b-eef6-4414-afbf-849f642211f4/kube-auth-proxy/0.log"
Sep 15 19:28:37.764697 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:37.764668    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-ht96l_8190ca1e-7e85-490e-b665-4712192e4d60/istio-proxy/0.log"
Sep 15 19:28:37.768489 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:37.768462    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-ht96l_8190ca1e-7e85-490e-b665-4712192e4d60/istio-proxy/1.log"
Sep 15 19:28:40.320956 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:40.320923    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-q87xv_87191718-cfa9-4ccd-98c9-d2dfd1470523/migrator/0.log"
Sep 15 19:28:40.341141 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:40.341114    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-q87xv_87191718-cfa9-4ccd-98c9-d2dfd1470523/graceful-termination/0.log"
Sep 15 19:28:41.769558 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:41.769528    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wvsmg_0dfbf315-3e43-45e4-9a05-4b64c8dbb36c/kube-multus-additional-cni-plugins/0.log"
Sep 15 19:28:41.788397 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:41.788369    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wvsmg_0dfbf315-3e43-45e4-9a05-4b64c8dbb36c/egress-router-binary-copy/0.log"
Sep 15 19:28:41.806260 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:41.806243    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wvsmg_0dfbf315-3e43-45e4-9a05-4b64c8dbb36c/cni-plugins/0.log"
Sep 15 19:28:41.824447 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:41.824423    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wvsmg_0dfbf315-3e43-45e4-9a05-4b64c8dbb36c/bond-cni-plugin/0.log"
Sep 15 19:28:41.842197 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:41.842175    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wvsmg_0dfbf315-3e43-45e4-9a05-4b64c8dbb36c/routeoverride-cni/0.log"
Sep 15 19:28:41.862807 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:41.862787    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wvsmg_0dfbf315-3e43-45e4-9a05-4b64c8dbb36c/whereabouts-cni-bincopy/0.log"
Sep 15 19:28:41.881234 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:41.881213    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wvsmg_0dfbf315-3e43-45e4-9a05-4b64c8dbb36c/whereabouts-cni/0.log"
Sep 15 19:28:42.077327 ip-10-0-129-227 kubenswrapper[2550]: I0915 19:28:42.077283    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rnhds_f07817f7-e8cf-4ba0-a211-9e3d85ee4f79/kube-multus/0.log"